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ABSTRACT 

Analysis  results  are  presented  from  the  application  of  new  DREV-developed  meth¬ 
ods  for  studying,  analyzing  and  assessing  the  value,  usefulness  or  goodness  of  the  tactical 
information  used  by  maritime  commanders  in  decision  making,  in  particular  for  over-the- 
horizon  targeting.  These  methods  relate  information  quality  and  system  measures  of  per¬ 
formance  (MOPs)  to  decision  and  mission  measures  of  effectiveness  (MOEs).  Finer-grained 
processing  of  relevant  exercise  data  than  has  been  used  in  previous  analysis  techniques 
results  in  hundredfold  increases  in  sample  sizes;  this  in  turn  improves  the  statistical  sig¬ 
nificance  and  accuracy  of  the  estimated  MOPs  and  MOEs.  DREV  model-based  measures 
(MBMs)  use  simple  decision  models  to  assess  the  value,  usefulness  or  goodness  of  the  pic¬ 
ture  a  commander  uses  for  a  decision  or  a  mission.  The  results  from  two  versions  of  DREV 
MBMs  are  compared. 


r£sum£ 

Les  resultats  d’analyses  presentes  decoulent  de  I’application  des  nouvelles  me- 
thodes  du  CRDV  pour  etudier,  analyser  et  evaluer  la  valeur,  I’utilite  ou  la  bonne  qualite 
de  I’information  tacbque  utilise  par  les  commandants  dans  leur  processus  de  d^ision  et 
particulierement  pour  supporter  le  ciblage  au-dela  de  I’horizon  de  ses  propres  capteurs. 
Ces  methodes  lient  la  qualite  de  I’information  ou  les  mesures  de  performance  des  sys- 
temes  (MDP)  aux  mesures  d’efficacite  (MDE)  des  decisions  prises  ou  des  missions  en  jeu. 
Un  traitement  plus  fin  de  toute  I’information  pertinente  d’un  exercice  militaire  permet 
d’obtenir  des  echantillons  dont  la  taille  est  accrue  par  un  facteur  de  I’ordre  du  centuple, 
ce  qiri  a  pour  effet  d’ameliorer  la  signification  statistique  et  la  pr^ision  des  MDP  et  MDE 
estim^s.  Les  nouvelles  mesures,  bas^s  sur  des  modeles  (MBM),  utilisent  des  modeles 
decisionnels  pour  determiner  la  valeur,  I’utilite  ou  la  bonne  qualite  de  I’image  tactique 
des  systemes  embarques  qui  traitent  I’information  maritime  utilise  par  un  commandant 
pour  une  decision  ou  une  mission.  Les  resultats  de  deux  versions  des  MBMs  du  CRDV  sont 
compares. 
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EXECUTIVE  SUAAAAARY 

Developing,  managing  and  sharing  a  common  tactical  picture  is  of  crucial  impor¬ 
tance  to  all  military  operations,  and  extended-range  anti-ship  missiles  and  over-the- 
horizon  sensors  that  support  over-the-horizon  targeting  (OTH-T)  now  impose  particularly 
demanding  requirements  for  a  wide-area  picture  (WAP)  in  the  maritime  environment. 
Producing  and  sharing  a  common  recognized  maritime  picture  (RMP)  of  high  enough  qual¬ 
ity  to  sustain  effective  military  operations  is  thus  one  of  the  Canadian  Navy’s  top  priori¬ 
ties. 


In  the  past,  the  effect  of  improvements  or  changes  to  communications,  command, 
control  and  information  system  (C3IS)  on  military  operations  or  mission  effectiveness  has 
been  evaluated  through  the  measurement  of  system  performance  and  information  quality 
quantities  knovm  as  measures  of  performance  (MOPs).  The  results  fall  short  of  demon¬ 
strating  clearly  the  impact  of  C3IS  improvements  or  changes  on  the  capability  to  conduct 
successful  operations  or  on  mission  effectiveness,  quantities  defined  here  as  measures  of 
effectiveness  (MOEs).  In  fact,  one  must  relate  information  quality  and  system  MOPs  to 
decision  and  mission  MOEs  in  a  causal  manner;  only  by  doing  so  can  one  establish  the  use¬ 
fulness  or  goodness  of  the  RMP  tactical  information  used  by  a  commander  to  plan  opera¬ 
tions  and  make  decisions. 

DREV  undertook  the  development  of  new  measures  as  the  Canadian  contribution 
to  the  AUS-CAN-NZ-UK-US  Ad  Hoc  Working  Group  on  the  "Management  of  Organic  and 
Non-organic  Information  in  a  Maritime  Environment"  (MONIME),  proposing  new  methods 
to  study,  analyze  and  assess  the  value,  usefulness  or  goodness  of  the  information  and  sys¬ 
tems  used  by  commanders  in  decision  making.  The  new  methods  provide  insights  not 
available  using  MONIME’s  current  methods  for  analyzing  a  series  of  live  or  simulated  ex¬ 
ercises.  Finer-grained  processing  of  the  raw  data  from  MONIME  studies  results  in  a  hun¬ 
dredfold  increase  in  sample  sizes,  thus  improving  statistical  significance  and  the  accuracy 
of  MOP  and  MOE  estimates.  The  new  DREV  MBAAs  were  presented  in  a  previous  document; 
here  we  show  how  they  assess  the  value,  usefulness  or  goodness  of  the  picture  a  com¬ 
mander  uses  for  decisions  or  missions  during  a  specific  exercise. 

The  MBMs  are  used  to  assess  the  data  collected  during  the  TIMSIM  *93  military  ex¬ 
ercise,  focusing  on  afloat  command  information  system  capabilities  in  support  of  over- 
the-horizon  targeting.  The  results  assess  the  value  of  the  information  made  available  to  a 
commander  by  considering  each  tactical  report  of  track  data  that  meets  a  given  set  of 
engagement  conditions.  Locational,  systems  and  temporal  data  contained  in  reports  are 
used  to  establish  possible  engagement  parameters  and  scenarios,  and  outcomes  subse¬ 
quent  to  all  decisions  that  are  feasible  at  that  moment  are  assessed  using  decision- 
process  model  definitions  and  algorithms  that  include  hit-probability  calculations  using 
ground-truth  information  (since  this  is  a  post-exercise  analysis).  Appropriate  areas  of  un¬ 
certainty  are  used  to  represent  the  intrinsic  level  of  uncertainty  of  missile-interception 
areas,  ground-truth  data  and  the  information  presented  by  C3I  systems  to  commanders. 
The  measures  include  reward  values  that  are  based  on  the  allegiances  of  contacts  in  the 
weapon  impact  area. 

The  results  from  the  new  MBAAs  are  also  compared  with  those  obtained  with  a  pre¬ 
vious  version  that  did  not  consider  the  cost  of  firing  a  weapon. 
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1  INTRODUCTION 

Providing  accurate  estimates  of  the  value,  usefulness  or  goodness  of  infor¬ 
mation  for  productive,  successful  dedsion  making  or  mission  planning  requires  that 
the  effects  of  changes  in  tactical  information  systems  and  procedures  on  mission 
effectiveness  be  determined.  Meeting  this  difficult  requirement  is  an  important 
step  in  the  cost-effective  development  and  integration  of  systems  to  support  mili¬ 
tary  operations,  and  the  assessment  of  man-in-the-loop  systems  holds  the  greatest 
potential  to  impact  the  effectiveness  of  our  forces. 

Improvements  to  v/ide-area  naval  tactical  picture  (WANTAP)  systems,  pro¬ 
cedures  and  standards  for  afloat  commanders  have  been  driven  by  the  need  for 
joint  and  coalition  operations  over  vride  geographical  areas  under  complex  geopo¬ 
litical  scenarios.  Extended  range  anti-ship  missiles  and  over-the-horizon  sensors 
that  support  over-the-horizon  targeting  (OTH-T)  nov/  lead  to  requirements  for  a 
wide-area  picture  (WAP),  in  geopolitical  situations  now  more  complex  than  those 
of  the  cold  war.  The  DREV  Work  Unit  "Data  fusion  rules”  was  established  to  ad¬ 
dress  the  requirement  defidendes  implied  by  these  new  "millennium”  missions 
and  theaters,  which  both  sea-  and  shore-based  national  systems  must  support. 

This  Work  Unit  is  based  upon  work  done  in  an  AUS-CAN-NZ-UK-US  C3 
(command,  control  and  communications)  Organization  (Refs.  1 ,  2)  work  program 
under  an  ad-hoc  working  group  (AHWG)  investigating  the  management  of  maritime 
information.  Results  and  data  from  a  series  of  experiments  for  analyzing  WANTAP 
systems  conducted  by  the  AHWG  are  used  to  evaluate  the  new  measures  developed 
by  DREV  for  assessing  the  impact  of  system  changes  on  mission  effectiveness.  The 
AUS-CAN-NZ-UK-US  C3  Organization’s  "Handbook  5  (HB5)  Guidelines  for  Maritime 
Information  Management”  (guidelines  for  the  procurement  of  national  C3I 
WANTAP-based  systems  for  compiling  and  sharing  accurate  WAPs)  (Refs.  2,  3),  a 
compendium  of  the  findings  and  recommendations  from  these  experiments,  is  used 
in  these  studies. 

The  new  model-based  measures  (MBMs)  developed  by  Defence  Research  Es¬ 
tablishment  Valcartier  (DREV)  were  designed  to  improve  the  accuracy  of  estimates 
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of  the  effect  of  information  quality  and  systems  performance  on  mission  effective¬ 
ness  through  the  application  of  a  set  of  dedsion  models  and  utility  functions.  Tra¬ 
ditional  evaluation  methodologies  reach  conclusions  by  examining  the  conse¬ 
quences  of  each  dedsion  actually  taken  during  an  exerdse  or  simulated  engage¬ 
ment.  Realistically,  very  few  significant  dedsions  can  actually  be  taken  under  such 
conditions,  so  conclusions  must  be  based  on  very  sparse  data.  The  new  DREV 
methodology  extends  the  statistical  base  by  considering  not  only  dedsions  that  are 
actually  made  but  also  those  that  m/ghf  have  been  made,  based  upon  the  informa¬ 
tion  available  to  the  commander  at  the  time.  By  modeling  all  potential  engage¬ 
ments,  vrith  due  consideration  to  the  errors  and  uncertainties  that  constrain  them, 
the  DREV  approach  provides  a  much  firmer  statistical  foundation  for  the  evaluation 
of  the  components  of  maritime  command  and  control  systems  than  do  the  meth¬ 
odologies  heretofore  used. 

These  dedsion  models  and  MBMs  were  reported  by  the  authors  in  Ref.  4,  but 
a  summary  is  presented  here  in  support  of  the  new  material  of  this  report.  Two 
software  implementations  of  these  measures  were  implemented  to  assess  engage¬ 
ment  opportunities  in  OTH-T  scenarios  so  to  reflect  the  expected  outcome  of 
eventual  engagement  operations  using  WANTAPs,  and  the  usefulness  of  these  soft¬ 
ware  tools  for  investigating  the  impact  of  WANTAP  picture  quality  and  system  per¬ 
formance  on  command  and  control  effectiveness  was  assessed. 

This  document  presents  analysis  results  of  the  application  of  DREV’s  MBM 
implementations  to  the  1993  Tactical  Information  Management  Simulation 
(TIMSIM  ’93)  data  (Refs.  5,  6).  Actual  models  are  presented  in  Refs.  4,  7.  More  de¬ 
tailed  information  concerning  a  previous  version  of  the  models,  along  vrith  the 
computer  implementation,  data  characterization,  preprocessing  and  earlier  TIMSIM 
’93  data  analyses  performed  with  that  version  are  described  in  Ref.  5.  The  new 
round  of  analyses  discussed  here  was  performed  in  order  to  upgrade  and  validate 
previous  conclusions  using  a  newer  and  better-tuned  model  implementation  whose 
latest  updates  are  documented  in  Refs.  4,  6-8.  Other  experimental  data  from  naval 
exerdses  (Refs.  9-11),  such  as  the  Rim  of  the  Padfic  1996  live  exerdse  (RIMPAC 
’96),  have  not  yet  been  processed  using  the  DREV  models,  but  results  from  analy¬ 
ses  of  these  experiments  are  used  to  validate  the  proposed  methodology. 
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Background  information  on  the  selected  DREV  MBMs  and  the  general  set-up 
used  to  analyze  live  or  simulated  military  exerdses  are  presented  in  Chapter  2. 
Descriptions  of  the  processing  applied  to  the  output  of  the  models  are  given  in 
Chapter  3.  Chapter  4  describes  supplementary  analyses  that  were  performed.  The 
results  are  discussed  in  Chapter  5.  Interpretation  of  the  model-based  measure  val¬ 
ues  and  conclusions  are  presented  in  Chapters  6  and  7  respectively.  A  glossary,  a 
list  of  acronyms  and  abbreviations,  and  summary  tables  of  the  analysis  results 
make  up  the  Appendices. 

This  work  was  performed  at  DREV  between  May  1995  and  June  1998  under 
Work  Unit  1ba16:  "Data  Fusion  Rules.” 
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2  MODEL-BASED  MEASURES  AND  ANALYSIS  SET-UP 

The  analyses  performed  are  based  on  the  dedsion  models  for  the  assess¬ 
ment  of  engagement  opportunities  defined  in  Refs.  4-7,  12,  using  the  computer 
implementation  described  in  Refs.  5,  6.  In  addition,  this  document  includes  several 
results  that  were  gathered  in  a  previous  round  of  analyses  using  supplementary 
analysis  programs  (see  Refs.  5,  6,  8),  which  are  not  part  of  the  models  as  such  but 
yield  insightful  complementary  information  on  delays  and  positional  errors. 

2.1  Summary  of  Model-based  Measures 

Essential  information  and  definitions  needed  to  understand  the  parameters 
used  in  the  analysis  set-up  are  presented  in  the  following  sub-sections.  Most  of  this 
material  is  addressed  in  more  detail  in  Ref.  4. 

2.1.1  Engagement  Situation 

Ships  within  an  area  of  operational  interest  (AOI)  displayed  by  a  WANTAP 
system  can  be  classified  according  to  their  perceived  or  reported  allegiance  as 
friendly  (F),  hostile  (H),  neutral  (N)  or  of  unknown  allegiance  (U),  a  subset  of 
NATO-assigned  allegiances.  Friendly  and  hostile  ships  are  military  vessels  of  the 
forces  in  conflict.  Usually  those  whose  allegiance  is  F  are  referred  to  as  the  "blue” 
force  and  those  whose  allegiance  is  H  as  the  "orange”  or  "red”  force.  Neutral 
ships-usually  merchant  ships,  liners  or  other  vessels  extraneous  to  the  conflict- 
are  often  represented  as  "green.”  The  allegiance  "unknown”  indicates  a  lack  of 
information  about  a  contact,  and  "white”  is  used  to  indicate  such  pending  tracks. 
A  perfect  reporting  system  with  all  the  appropriate  information  would  not  need 
this  classification.  Only  F,  H  and  N  allegiances  appear  in  the  ground-truth  (GT). 

In  our  models,  an  engagement  situation  occurs  every  time  an  armed  ship 
from  the  blue  force  has  knowledge  of  the  presence  of  an  enemy  ship  (orange  force) 
within  range  of  its  weapons.  A  typical  such  situation  occurs  every  time  the  com¬ 
mander  of  an  armed  ship  (CS)  selects,  for  the  next  engagement,  an  intended  hos¬ 
tile  (IH)  target  from  those  displayed  by  the  CS  tactical  system.  We  assume  that  the 
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FIGURE  1  -  Positional  errors  with  and  without  prediction  relative  to  GT  posi¬ 
tions  at  three  times 

commander  follows  appropriate  procedures,  that  the  target  is  located  within  the 
physical  boundaries  that  limit  the  systems  (weapons,  sensors,  information  and 
command)  available  to  the  CS  and  that  it  can  be  engaged  under  the  rules  of  en¬ 
gagement  (ROEs)  in  effect  at  that  time. 

2.1. 1.1  Contact  Information  Reports 

The  tactical  information  about  a  ship  forwarded  to  a  CS  through  different 
information  channels  and  means  contains  the  identification  (class-name)  of  the 
ship,  its  position  and,  eventually,  other  information  such  as  course,  speed  and  al¬ 
legiance.  Each  occurrence  of  such  combined  data  is  referred  to  as  an  information 
report  on  a  contact.  The  "position  time”  tpos  (to  in  Fig.  1)  indicates  the  time  at 
which  the  information  was  acquired  (sensor  time).  The  "report  time”  tr  (ti  in 
Fig.  1)  is  the  time  when  the  information  report  was  made  available  to  its  redpi- 
ent’s  database.  In  a  real  situation  another  time  may  be  necessary:  the  com¬ 
mander’s  dedsion  time  (ta  in  Fig.  1),  which  occurs  after  ti.  Figure  1  shows  a  first- 
order  position  prediction  algorithm  such  as  dead  reckoning  (DR),  with  the  reported 
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position  at  to,  the  predicted  positions  at  ti  and  t2,  and  the  GT  positions  at  to,  ti  and 
t2.  With  respect  to  the  preceding  definition  of  contact  reports,  we  can  consider 
that  an  engagement  situation  occurs  each  time  the  commander  of  an  armed 
ship  from  the  blue  force  receives  an  information  report  on  a  presumably  hos¬ 
tile  contact.  For  the  MBMs,  hostility  depends  on  class-name  identification  or  alle¬ 
giance  as  indicated  in  the  information  report. 

2.1.2  Ground  Truth 

An  ultimate  goal  in  C3I  is  to  share  the  same  WANTAP  simultaneously  by  all 
units  of  a  battle  forced  In  real  systems,  however,  the  WANTAP  view  or  tactical  in¬ 
formation  available  to  commanders  may  be  incomplete,  erroneous  and  cluttered 
vnth  duplicated  information.  In  contrast  to  this  partial  or  incorrect  view  there  is, 
of  course,  only  one  actual  wide-area  naval  situation  at  any  given  time:  the  ground 
truth  or  GT.  GT  information  consists  of  identification  and  allegiance  of  each  ship, 
along  with  its  location  (speed  and  course  may  be  included  or  can  be  estimated 
from  the  sequence  of  position  values  in  the  GT)  at  any  given  time^  for  the  entire 
area  to  be  controlled  by  a  force  over  the  period  of  time  considered.  Thus,  the  GT 
WAP  can  be  viewed  as  a  list  of  position  reports  across  a  time  scale  and  a  geo¬ 
graphical  area  with  the  required  granularity.  If  necessary,  it  can  be  reconstructed 
by  combining  all  the  information  collected  during  a  life  exerdse,  but  a  recon¬ 
structed  GT  is  not  as  complete  and  accurate  as  one  used  to  drive  a  simulated  exer¬ 
dse  (see  GT  error  rate  in  Refs.  9,  10). 


^  This  view  was  originally  defined  by  the  Canadian  Navy  as  the  common  Recognized  Maritime  Picture  (RMP), 
but  in  HB5  it  is  now  defined  as  the  Maritime  Tactical  Picture  (MTP):  the  global  picture. 

2  In  practice,  the  GT  data  may  be  listed  according  to  a  discrete  time  variable  with  short  steps,  often  one 


minute. 
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FIGURE  2  -  A  typical  engagement  situation  with  a  weapon  aiming  vector  and  a 
weapon-uncertainty  area  centered  at  the  intended  hostile  (IH)’s  posi¬ 
tion 

2.1.3  Positional  Uncertainty 

The  positional  information  in  WANTAP  systems  is  uncertain  for  a  number  of 
reasons.  This  lack  of  predsion  is  expressed  by  an  area-of-uncertainty  (AOU):  there 
is  a  probability  Pi  that  the  object  lies  anywhere  within  this  area  and  a  probability 
Po  =  1  -  Pi  that  it  lies  outside.  Typically,  a  system  may  use  a  confidence  level  of 
95%;  that  is,  the  probability  of  the  object  being  outside  by  chance  only  is  0.05.  In 
some  systems  the  contact  AOU  is  provided  by  the  source  of  data,  but  since  this  in¬ 
formation  is  not  available  for  all  the  contact  reports  used  in  our  analysis,  we  have 
simplified  the  computation  by  assuming  that  po  is  null  or  negligible  (i.e.,  Pi  =  1). 
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2.1.3.1  Weapon-uncertainty  Area 

In  DREV  MBMs  the  weapon-uncertainty  area  (WUA),  also  called  the  weapon 
footprint,  is  modeled  as  an  ellipse  centered  on  the  location  at  which  the  weapon  is 
aimed.  Its  orientation,  i.e.,  its  major  axis,  lies  along  a  straight  line  connecting  the 
weapon-launching  ship  to  the  presumed  location  of  the  intended  target  (see 
Fig.  2).  The  WUA  is  specified  by  three  parameters  : 

1 .  the  half-length  of  the  minor  axis  of  the  ellipse  ; 

2.  the  half-length  of  the  major  axis  of  the  ellipse;  and 

3.  a  weight  function  defined  over  the  area  of  the  ellipse. 

The  weight  function,  defined  over  all  points  vrithin  the  elliptic  area  (Fig.  2), 
assigns  to  each  such  point  a  weight  indicating  the  likelihood  of  that  point’s  being 
hit  by  the  weapon  if  it  is  launched.  Any  suitable  function  can  be  used  for  this  pa¬ 
rameter. 

The  scenario  of  Fig.  2  can  be  modified  if  it  is  assumed  that  the  commander 
uses  predicted  positions,  which  may  be  calculated  by  his  information  systems. 
Then  at  ti  instead  of  using  the  report  at  time  ti  (Rt^)  of  Fig.  1,  he  uses  the  pre¬ 
dicted  position  at  time  ti  (Pt ).  This  new  scenario  with  the  predicted  position  of 
the  intended  hostile  (IH-P)  is  shown  in  Fig.  3. 

2.1.3.2  Circular-uncertainty  Area 

A  drcular-uncertainty  area  (CUA)  represents  a  possible  positional  error  in  a 
contact  information  report  from  a  given  WANTAP  system.  The  CUA,  which  is 
equivalent  to  the  AOU  of  the  spedfied  contact  information  report,  must  be  applied 
to  all  the  GT  positional  reports  used  to  assess  a  given  measure  at  decision  time. 
Figure  4  shows  a  CUA  for  each  GT  position  at  a  given  time-except  for  the  com¬ 
mander’s  ship,  which  is  used  as  a  reference.  The  CS  displayed  information  may  not 
contain  any  data  for  a  given  GT  position  or  it  may  display  "phantom”  ships.  The 
displayed  data  may  also  contain  other  errors-in  type  of  contact  or  allegiance,  for 
example. 
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i 

CS  -  commander’s  ship 

H  'H  -  intended  hostile  target 

IH-P  •  IH  predicted  position 

H  -  hostile 

F  -  friend 

N  -  neutral 

U  -  unknown 


FIGURE  3  -  A  typical  engagement  situation  with  a  weapon  aiming  vector  and  a 
weapon-uncertainty  area  centered  at  the  IH-P’s  position 


Within  the  models,  a  CUA  is  spedfied  by  two  parameters: 

1 .  a  drcle  radius;  and 

2.  a  probability  distribution  function  (pdf)  over  the  circular  area. 

The  pdf  parameter  is  a  probability  density  function  defined  over  all  points 
within  the  drcle.  It  assigns  to  each  point  a  probability  that  the  ship  (if  there  is 
one)  corresponding  to  the  contact  information  report  is  at  that  point  at  the  posi¬ 
tion  time  indicated  in  the  report  (or  at  a  later  time,  to  include  the  effect  of  delay 
on  this  probability).  Although  any  bidimensional  pdf  can  be  used  for  this  parame¬ 
ter,  we  use  uniform,  Gaussian  and  triangular  bivariate  distributions  as  starting 
points,  as  described  in  the  next  subsection.  The  Gaussian  bivariate  pdf  is  defined 
over  the  entire  plane,  so  a  truncated  version-one  with  a  zero  value  outside  the 
CUA  (Ref.  4)-is  used  to  reduce  the  computation  load. 
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FIGURE  4  -  A  CUA  is  applied  to  each  GT  position,  except  for  the  reference  ship 


A  CUA  can  be  assigned  to  any  contact,  but  in  the  particular  case  of  a  pre¬ 
sumed  hostile  contact  we  call  it  instead  a  target-uncertainty  area^  (TUA),  since  we 
are  primarily  concerned  with  the  impact  and  outcome  of  engagement  decisions 
concerning  contacts  of  this  type. 

The  CUA  represents  not  only  the  expected  errors  found  in  operational  sys¬ 
tems  but  also  errors  within  GT  position  reports.  It  has  been  observed  (Refs.  9,  10) 
that  even  in  controlled  experiments  proper  GT  data  files  may  have  to  be  recon¬ 
structed  from  partial  information:  holes  in  the  GT  data  may  have  been  filled  by 
interpolation  that  introduces  positional  impredsion.  In  some  cases,  parts  of  the  GT 
files  may  even  be  reconstructed  using  contact  information  reports  from  a  com¬ 
mander’s  database.  In  addition,  it  cannot  be  assumed  that  all  positions  in  WANTAP 
contact  information  reports  are  matched  by  equivalent  GT  position  reports  at  the 
exact  same  times.  In  such  cases  we  have  to  interpolate  between  two  GT  position 


^  In  Ref.  5,  we  do  not  use  the  TUA  acronym  and  refer  only  to  the  more  general  CUA. 


UNCLASSIFIED 

11 


reports  to  get  the  equivalent  GT  position  at  a  particular  time.  All  these  considera¬ 
tions  can  be  combined  in  the  view  that  a  CUA  around  any  GT  position  report  repre¬ 
sents  a  combination  of  all  the  errors  in  both  the  CS  contact  information  and  GT  po¬ 
sition  reports.  This  greatly  simplifies  the  computation  required  for  each  MBM. 

Finally,  we  will  later  show  (Section  3.6)  that  considering  CUAs  around  GT 
positions  accounts  for  all  possible  WUA  center-position  alignments  within  a  given 
TUA  (the  CUA  of  a  target).  The  uncertainty  of  the  TUA  is  therefore  transferred  and 
blended  into  an  equivalent  positional  uncertainty  around  the  GT  location  values  in 
the  vidnity  of  the  WUA.  This  allows  computations  in  the  models  to  use  a  single 
WUA  for  a  given  target  and  still  accounts  for  all  the  uncertainty  involved. 

2.1.4  Engagement-opportunity  Reference  Times 

The  previous  section  defined  an  engagement  situation  as  occurring  when¬ 
ever  the  commander  of  an  armed  blue  ship  receives  an  information  report  on  a 
presumed  hostile  contact.  That  information  report  includes  a  position  time  to  and  a 
report  time  ti,  with  to<ti.  The  models  we  propose  may  use  either  of  these  two 
time  values  as  the  actual  time  of  engagement  opportunity,  i.e.,  the  time  at  which 
an  engagement  might  take  place  (or  might  have  occurred).  Of  course,  in  reality,  an 
engagement  dedsion  cannot  be  taken  before  the  existence  of  the  information  re¬ 
port  is  known.  However,  allowing  the  selection  of  different  times  of  engagement  in 
the  models  yields  essential  measures  for  estimating  the  impact  of  system  changes 
on  mission  effectiveness. 

2.1.5  Measures  of  Engagement  Opportunities 

In  geopolitical  conflicts  where  engagement  situations  may  arise,  it  is  of  cru- 
dal  importance  to  evaluate  the  impact  of  an  offensive  action  such  as  the  launch  of 
a  weapon.  This  impact  may  be  evaluated  according  to  many  parameters  and  from 
many  different  points  of  view:  for  example,  the  cost  and  effort  of  conducting  an 
offensive  action,  expected  enemy  damage  as  a  result  of  such  action,  tactical  con¬ 
sequences,  and  geopolitical  and  other  aspects.  Obviously,  the  impact  is  not  neces¬ 
sarily  the  same  for  the  personnel  aboard  a  ship  under  threat  in  a  short-range  en¬ 
counter  as  it  is  for  the  fleet  commander  or  the  government  of  the  nation  under 
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whose  flag  the  ship  sails.  It  is  thus  unreasonable  to  hope  to  identify  a  simple  meas¬ 
ure  of  impact  that  would  apply  to  all  cases  of  interest,  but  it  is  surely  possible  to 
capture  an  essential  component  of  that  impact  by  establishing  some  quantification 
of  the  outcome  of  an  offensive  action.  Two  measures  are  proposed  here  for  this 
purpose:  the  "pertinence-of-engagement”  and  the  "intended-target-opportunity.” 
Their  descriptions  follow. 

2.1.5.1  Pertinence-of-engagement  Measure 

The  pertinence-of-engagement  (POE)  measure  is  a  means  to  assess  the  gen¬ 
eral  outcome  of  launching  a  weapon  at  a  target  in  an  engagement  situation.  It  can 
be  viewed  as  a  measure  of  the  pertinence  of  the  dedsion  to  engage.  It  involves 
three  aspects: 

1.  The  cosH-of-firing  (COF)  the  weapon:  a  negative  quantity  expressing 
the  expense  (economic,  material  and  personnel)  assodated  vrith  the 
launch  of  the  weapon. 

2.  The  hit  probability^  (HP):  the  probability  that  the  weapon  vrill  hit 
something  (a  ship)  within  its  WUA.  It  is  computed  by  looking  at  the 
joint-probability  distribution  of  all  the  ships  whose  CUA  intersects  the 
WUA  at  the  time  of  engagement  opportunity  (see  Ref.  5  for  precise 
mathematical  formulations  and  computing  algorithms). 


In  this  context  the  cost  is  taken  in  its  niathematicat  sense,  i.e.,  a  utility  function/value,  as  is  commonly 
used  in  mathematical  models. 

5  In  Ref.  5  two  different  hit  probabilities  are  defined:  the  General  Hit  Probability  (GHP)  and  the  Specific  Hit 
Probability  (SHP).  It  has  been  observed  that  this  technique  could  lead  to  unrealistic  assessments  of  outcomes 
in  some  cases,  i.e.,  when  the  CUA  of  the  intended  target  does  not  intersect  the  WUA,  and  hence  its  SHP  is  0, 
while  the  CUAs  of  a  number  of  friendly  or  neutral  ships  do  intersect  it.  The  models  now  consider  only  the 
GHP  (called  simply  HP).  The  probability  of  hitting  a  specific  ship  is  still  computed  and  available  within  the 
models,  but  it  is  integrated  in  the  reward-value  terms  of  the  equations  to  avoid  the  absorbing  effect  of  mul¬ 
tiplying  by  a  zero  value,  thus  preserving  the  continuity  of  the  measures. 
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FIGURE  5  -  Intersections  of  three  CUAs  with  the  WUA:  surface  overlaps 


3.  The  pertinence-reward-value^  (PRV),  which  combines  the  allegiances 
of  the  potential  target  ships  with  their  probability  of  actually  being  hit 
should  the  weapon  be  fired,  for  all  ships  within  the  WUA  at  the  time 
of  the  engagement.  See  Ref.  12  for  explanations  and  Refs.  5,  6  for  the 
predse  mathematical  formulations  and  computing  algorithms. 

Figure  5  shows  the  CS  aiming  at  the  last  reported  position  of  the  intended 
hostile  (IH)  target.  Black  text  symbols  represent  what  is  shown  on  the  CS  tactical 
display  and  white  text  symbols,  with  their  surrounding  CUAs,  represent  the  actual 
GT  positions  of  ships  superimposed  over  the  CS  tactical  display  area  at  the  same 
time.  A  MBM  hit  probability  is  defined  by  the  joint  probability  distribution  of  all 
ships  whose  CUA  intersects  the  WUA;  e.g.,  the  surface  overlaps  of  two  N  GT  and 
the  IH  GT  CUAs  in  the  snapshot  shown  in  Fig.  5. 


^  The  PRV  is  equivalent  to  the  General  Reward  Value  (GRV)  in  Ref.  5. 
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2. 1.5.2  Intended-target-opportunity  Measure 

Though  quite  similar  to  the  POE  measure,  the  intended-target-opportunity 
(ITO)  measure  is  more  specific  because  it  focuses  on  the  outcome  of  launching  the 
weapon  with  respect  to  the  intended  target  in  the  engagement  situation.  It  in¬ 
volves: 

1 .  The  COF  (as  defined  previously), 

2.  the  HP  (as  defined  previously)  and 

3.  the  target- reward- value^  (TRV),  which  combines  the  allegiances  of 
ships  v/ith  their  probability  of  actually  being  hit,  should  the  weapon 
be  fired,  for  all  the  ships  within  the  WUA  at  the  time  of  the  engage¬ 
ment  opportunity.  It  is  similar  to  the  PRV,  except  that  it  gives  lower 
weight  to  hostile  ships  other  than  the  intended  target  (see  Ref.  12  for 
explanations  and  Refs.  5,  6  for  the  predse  mathematical  formulations 
and  computing  algorithms). 

2.1.6  Model-based  Measures 

The  various  combinations  of  the  two  measures  of  engagement  opportunities 
with  the  time  at  which  engagement  might  take  place  lead  us  to  consider  six  differ¬ 
ent  models  for  the  assessment  of  engagement  opportunities.  Each  provides  a  nu¬ 
merical  measure  that  assesses  the  outcome  of  an  offensive  action,  i.e.,  launching  a 
weapon,  under  a  certain  point  of  view  (POE  or  ITO  measure)  and  in  a  certain  situa¬ 
tion  (time  of  engagement  opportunity  with  and  without  positional  prediction®). 


7  The  TRV  is  equivalent  to  the  Specific  Reward  value  (SRV)  in  Ref.  5 

®  Algorithms  such  as  dead-reckoning  (DR)  estimation  are  commonly  used  as  navigation  aids.  First-order  DR 
estimates  the  position  of  a  moving  body  from  the  last  values  of  its  reported  position  and  velocity  vector. 
Higher-order  DR  uses  other  information  such  as  displacement  influences  and  angular  accelerations. 
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FIGURE  6  -  CS  display  of  two  possible  positions  of  the  intended  target 

These  measures  causally^  determine  the  impact  of  information  lacks  and  system 
limitations  on  mission  effectiveness. 


Figure  6  shows  a  possible  set  of  engagement  conditions  for  two  target- 
position  values  displayed  to  the  commander:  with  and  without  positional  predic¬ 
tion.  The  positional  prediction  illustrated  is  first-order  dead  reckoning,  which  as¬ 
sumes  constant  speed,  no  change  of  heading  and  no  secondary  effects  such  as 
variations  caused  by  wind  or  water  currents.  MBMs  can  assess  different  hit  prob¬ 
abilities  even  when  the  information  displayed  to  the  CS  is  produced  using  different 
techniques  and  processing  algorithms. 


^  In  causal  analysis,  the  probable  causes  of  observed  events  (a  productive  hit  of  a  hostile  ship  or  a  counter¬ 
productive  hit  of  a  friendly  ship)  are  traced,  while  causality  is  a  relation  between  simple  results  and  the 
state  of  the  world.  This  includes  environment,  platforms,  information  quality  and  system  states.  Here,  it  is 
more  a  question  of  correlation  between  the  result  of  a  decision  and  the  quality  of  information  as  well  as  the 
performance  of  the  systems  involved  in  the  closed-loop  scheme  of  command  and  control. 
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2. 1.6.1  Model  1  -  Pertinence-of-engagement  without  Delay 

The  "pertinence-of-engagement  without  delay"  model  assumes  no  delay  be¬ 
tween  the  time  the  information  is  acquired  by  the  sensor,  acquisition  device  or 
mechanism  and  the  time  this  information  is  made  available  as  an  information  re¬ 
port  to  the  commander  of  each  blue  ship.  That  is,  we  assume  that  position  time  is 
equal  to  report  time  and  that  the  engagement  might  have  taken  place  at  position 
time.  With  respect  to  information  degradation  over  time,  the  "ideal-engagement- 
situation”  (lES)  model  serves  a  standard  to  compare  other  models  with.  The  POE 
measure  output  of  the  model  is  defined  as  a  POE-optimal  (POE-0)  measure,  and 
lES  POE-0  measures  can  be  used  as  a  baseline  to  assess  the  dependence  of  alter¬ 
nate  engagement  situations  on  system  performance.  This  is  the  "sensor”  baseline, 
since  the  systems  primarily  involved  are  the  sensors  themselves  with  their  geo- 
referenced  results;  other  systems  have  not  yet  had  the  opportunity  to  affect  the 
value  of  reports  for  decision  making. 

2.1.6.2  Model  2  -  Pertinence-of-engagement  with  Delay  and  without  Predic¬ 
tion 

The  "pertinence-of-engagement  with  delay  and  vrithout  prediction”  model 
assumes  that  an  engagement  may  take  place  only  at  report  time  (with  report  time 
>  position  time).  It  can  be  applied  to  each  engagement  opportunity.  The  resulting 
measure  is  defined  as  a  POE-delay  (POE-D)  measure,  indicating  a  delay  between 
information  acquisition  and  the  occurrence  of  the  engagement  situation.  The  in¬ 
formation  is  not  updated  during  this  delay  and  the  target  location  used  is  the  loca¬ 
tion  recorded  at  position  time,  so  the  model  deals  with  a  time-degraded  tactical 
picture. 

2.1.6.3  Model  3  -  Pertinence-of-engagement  with  Delay  and  Prediction 

Like  the  previous  measure,  the  "pertinence-of-engagement  with  delay  and 
prediction”  model  also  outputs  a  POE  value:  a  POE-delay-prediction  (POE-D-P) 
measure.  It  assumes  that  engagement  may  take  place  only  at  report  time  (vrith  re¬ 
port  time  >  position  time),  but  allows  for  the  use  of  some  means  to  extrapolate  or 
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predict  a  new  target  location  at  the  time  of  engagement  opportunity.  In  this 
model,  the  weapon  is  aimed  at  the  predicted  location  of  the  target  at  report  time. 

2.1.6.4  Model  4  -  Intended-target-opportunity  without  Delay 

The  "intended-target-opportunity  without  delay”  model  is  similar  to  Model 
1,  except  that  it  outputs  an  ITO  measure,  defined  as  an  ITO-optimal  (ITO-0)  meas¬ 
ure,  instead  of  a  POE-0  measure.  Compared  to  the  POE  measure,  the  ITO  measure 
assigns  a  lower  reward  value  for  damaging  an  enemy  ship  that  is  not  the  intended 
target. 

2.1.6.5  Model  5  •  Intended-target-opportunity  with  Delay  and  widiout  Predic¬ 
tion 

The  "intended-target-opportunity  with  delay  and  without  prediction”  model 
is  similar  to  Model  2,  except  that  it  outputs  an  ITO  measure,  defined  as  an  ITO- 
delay  (ITO-D)  measure,  instead  of  a  POE-D  measure. 

2. 1.6.6  Model  6  -  Intended-target-opportunity  with  Delay  and  Prediction 

The  "intended-target-opportunity  with  delay  and  prediction”  model  is  simi¬ 
lar  to  Model  3,  except  that  it  outputs  an  ITO  measure  and  is  defined  as  an  ITO- 
delay-prediction  (ITO-D-P)  measure,  instead  of  a  POE-D-P  measure  using  the  pre¬ 
dicted  position  at  the  WANTAP  report  time. 

2.1.7  Meaning  and  Usefulness  of  these  Models 

Models  1  and  4  may  be  viewed  as  "optimal”  models  since  they  are  equiva¬ 
lent  to  assuming  that  information  reports  are  available  instantaneously,  as  soon  as 
information  is  gathered  by  sensors  or  acquisition  devices  or  mechanisms  (i.e.,  posi¬ 
tion  time  =  report  time  =  time  of  engagement  opportunity).  These  models  deliver 
the  maximum  usefulness  value  of  the  information  that  can  be  provided  to  a  com¬ 
mander.  This  value  is  the  source  or  sensor  baseline  that  can  be  used  to  compare 
the  impact  of  changes  to  system  architecture  on  mission  effectiveness. 
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Models  2  and  5  are  "time  degraded”  models  in  which  target  information  has 
not  been  updated  since  position  time  (i.e.,  position  time  <  report  time  =  time  of 
engagement  opportunity)'' o.  Time  degradation  of  the  information  represents  sys¬ 
tem  limitations  that  are  assumed  to  imply  sub-optimal  performance. 

Models  3  and  6  are  "predictive”  models  in  the  sense  that  they  try  to  com¬ 
pensate  for  time  degradation  by  updating  information  according  to  a  prediction 
algorithm.  Any  prediction  scheme  can  be  fitted  into  these  models.  Comparing  the 
results  obtained  using  these  models  to  those  obtained  using  "time  degraded”  mod¬ 
els  (i.e.,  Models  2  and  5)  may  be  helpful  in  order  to  estimate  the  quality  of  a  given 
predictive  algorithm  and  to  assess  its  impact  on  mission  effectiveness.  As  a  starting 
point,  we  use  a  predictive  algorithm  that  assumes  dead-reckoning^^  of  target  ships. 

Models  2  and  5  and  Models  3  and  6  can  be  extended  to  become  Models  2* 
and  5*  and  Models  3*  and  6*  by  taking  into  account  the  time  a  commander  takes  to 
deliberate  and  to  reach  a  decision,  at  time  t:  in  Fig.  1.  Models  1  and  4  use'll  to. 
Models  2  and  5  and  Models  3  and  6  use  ti  and  Models  2*  and  5*  and  Models  3*  and  6* 
use  the  third  time:  t2  >  ti  >  to.  The  differences  between  the  results  from  these 
models  can  be  used  in  the  relative  assessment  of  the  role  of  the  human  in  the  de- 
dsion  process  or  in  the  information  and  other  systems  involved.  Such  assessments 
can  be  used  in  training  to  compare  and  rank  the  performance  of  dedsion  makers 
(this  extension  has  not  yet  been  done  but  is  planned  as  a  future  exploitation  of 
DREVMBMs). 


The  position  time  or  sensor  detection  time  is  earlier  than  the  report  time.  We  set  the  time  of  an  engage¬ 
ment  to  the  report  tinr^e  so  as  to  measure  the  data-aging  effect  on  the  result  of  the  engagement. 

Dead-reckoning,  "the  determining  of  the  position  of  a  vehicle  at  one  time  vrith  respect  to  its  position  at  a 
different  time  by  application  of  vectors  representing  courses  and  distances”,  Ref.  13. 

The  sensor  time  to  is  the  same  for  any  model  for  a  given  report.  The  other  times,  ti  and  t2,  are  the  same 
for  the  various  models  as  long  as  they  refer  to  the  same  database  and  report.  For  the  same  report  but  from  a 
different  database,  e.g.,  for  another  ship,  ti  and  t2  may  be  different. 
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The  dedsion  rule  conveyed  within  these  models  is  implidt,  since  the  output 
measure  (POE  or  ITO)  is  a  direct  assertion  about  the  outcome  of  a  commander’s 
engagement  dedsion.  The  actual  range  of  values  taken  by  the  POE  and  ITO  meas¬ 
ures  is  implementation-dependent  (according  to  the  values  set  for  allegiance  fac¬ 
tors  in  the  PRV  or  TRV  and  the  utility  imposed  by  the  COF)  but  it  is  a  continuous 
interval  from  a  lower  bound  indicating  the  worst  possible  outcome  to  an  upper 
bound  indicating  the  best.  From  the  point  of  view  of  the  commander  of  a  blue 
force  ship,  the  worst  possible  outcome  of  an  engagement  is  a  sure  hit  on  a  friendly 
or  neutral  platform.  The  best  outcome  is  a  sure  hit  on  the  intended  hostile  target 
or  on  any  hostile  target. 

The  following  results  are  based  on  the  application  of  "Configuration  4” 
(Ref.  4):  Multiple  engagements/multiple  models.  Two  or  more  models  are  selected 
and  applied  concurrently  over  a  series  of  engagement  situations,  thus  allowing  a 
comparative  analysis  of  WANTAP  system  results  from  any  sample  of  engagement 
situations.  Conclusions  drawn  from  an  analysis  with  this  configuration  are  statisti¬ 
cally  significant  when  the  sample  size  is  large  enough,  say  in  excess  of  one  hun¬ 
dred. 


This  technique  can  be  used  effectively  to  assess  the  impact  of  the  delay  be¬ 
tween  information  acquisition  and  information  availability  (the  difference  between 
the  position  time  and  the  report  time  in  the  contact  information  reports),  of  the 
use  of  position-prediction  algorithms  on  engagement  outcomes  and  of  system  and 
information  changes  on  the  quality  of  decisions  and  on  mission  effectiveness.  This 
is  achieved  by  concurrent  applications  of  POE  models  (Models  1,  2  and  3)  and/or 
ITO  models  (Models  4,  5  and  6)  over  the  same  engagement  situation  (or  the  same 
set  of  engagement  situations)  and  comparative  evaluation  of  the  output  measures. 

In  the  analyses,  the  set  of  scenarios  is  applied  consecutively  to  large  sam¬ 
ples  of  engagement  situations  from  a  simulation  experiment  (TIMSIM  ’93).  Actual 
engagement  dedsions  are  relatively  few  in  such  experiments  (as  they  should  be  in 
live  or  simulation  exerdses)  so  the  conclusions  drawn  from  their  outcomes  have 
very  poor,  if  any,  statistical  significance.  In  contrast,  applying  the  models  as  we 
did  over  the  experiment  data  yields  samples  of  sizes  approximately  a  hundredfold 
larger,  which  strengthens  the  statistical  soundness  of  the  inferences  drawn.  During 
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the  four  days  of  this  exerdse  we  observed  33  qualified  engagements,  as  compared 
to  3882  MBM-simulated  ones  for  the  Force  Over-the-horizon  Track  Coordinator 
(FOTC)  alone,  a  factor  of  117  (for  more  information  about  this  aspect  see 
Refs.  5,  12,  14,  15). 

2.2  Analysis  Set-up 

The  analyses  performed  are  based  on  the  dedsion  models  for  the  assess¬ 
ment  of  engagement  opportunities  defined  in  Ref.  4  and  summarized  above.  We 
assume  a  memory-less  dedsion-action  scheme  scenario  that  jumps  from  one  en¬ 
gagement  opportunity  to  the  next. 

In  a  closed-loop  simulation,  sequential  events  would  affect  future  events 
and  another  module  would  have  to  be  added  to  the  current  implementation  of  the 
MBMs,  or  in  the  determination  of  the  value  of  information  in  dedsion  making,  to 
assess  the  damage  to  a  target  given  a  particular  engagement  outcome  or  after  one 
or  several  hits  to  a  contact. 

Here  we  focus  on  the  memory-less  case  and  assume  that  joint  probabilities 
of  WUA  and  CUA  intersections  are  found  using  numerical  techniques,  either  brute- 
force  Monte  Carlo  or  integral  approximations.  We  used  the  computer  implementa¬ 
tion  of  the  MBMs  as  described  in  Refs.  5,  6.  In  addition,  this  document  includes 
several  results  that  were  gathered  in  a  previous  round  of  analyses  using  supple¬ 
mentary  analysis  programs  that  are  not  part  of  the  models  as  such,  but  yield  com¬ 
plementary  information  on  delays  and  position  errors. 

The  decision  models  were  set  up  in  an  analysis  framework  consisting  of  sev¬ 
eral  simulation  runs  according  to  a  number  of  different  scenarios.  With  respect  to 
the  six  MBM  decision  models,  each  simulation  run  applies  two  models-Models  1 
and  4,  Models  2  and  5  or  Models  3  and  6- to  the  same  set  of  data.  Each  run  outputs 
both  the  pertinence-of-engagement  (POE)  measure  and  the  intended-target- 
opportunity  (ITO)  measure  for  an  otherwise  identical  model  set-up  for  delay  and 
prediction.  Each  run  processes  a  data  set  consisting  of  the  ground-truth  (GT)  data 
for  one  day  of  the  TIMSIM  '93  simulation  and  the  data  from  one  WAPS  (wide-area 
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TABLE  I 

Data  sets  used  in  the  DREV  MBM  demonstration 
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picture  systems  such  as  JOTS,  TWCS,  JMCIS  and  GCCS)13  for  that  day;  applying  the 
models  in  the  multiple  engagements/multiple  models  configuration  for  each  run. 
Thus,  four  days  of  TIMSIM  '93  data  yield  16  distinct  data  sets  as  identified  in  Ta¬ 
ble  1. 


The  Joint  Operational  and  Tactical  System,  Tomahawk  Weapons  Control  System,  Joint  Maritime  Command 
Information  System,  and  Global  Command  and  Control  System  respectively. 

The  OTCIXS,  Officer  In  Tactical  Command  Information  Exchange  System,  or  ACIXS,  Allied  Command  Infor¬ 
mation  Exchange  System,  Is  a  communications  systems  that  usees  satellite  technologies  at  data  rates  ranging 
from  2  400  to  9  400  kb/s.  TADIXS,  Tactical  Data  Information  Exchange  System,  Is  the  real  system  (UHF 
SATCOM  data  link)  and  OTCIXS  or  ACIXS  Is  a  concept. 
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TABLE  II 

Mapping  of  the  (16X6)  96  distinct  model  outputs 
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In  this  context  the  FOTC  compiles  the  tactical  track  data  (manages  the  sur¬ 
face,  sub-surface  and  air  tracks)  and  sends  its  tactical  picture  to  two  Surface  Ac¬ 
tion  Groups  (SAG1  and  SAG2).  The  TWCS  performs  a  similar  information- 
management  function  and  tactical-picture  compilation  tailored  to  the  needs  of 
OTH-T,  since  it  is  a  weapon  control  station  for  this  purpose. 


In  the  Day  1  and  2  results  of  Table  I  the  hostile  forces  were  cooperative; 
that  is,  they  used  active  devices  such  as  radar  and  radios.  On  the  other  days  they 
were  non-cooperative. 

Three  simulation  scenarios  were  executed  for  each  of  these  data  sets,  cor¬ 
responding  to  the  three  pairs  of  applied  decision  models.  Consequently  we  ob¬ 
tained  a  total  of  48  different  simulation  runs  with  96  distinct  model  outputs,  48 
POE  and  48  ITO  sets  of  results.  These  scenarios  and  model  numbers  are  mapped  in 
Table  II. 


Each  run  used  an  identical  set  of  simulation  parameters,  shown  in  Table  III. 
All  the  engagement  situations  occurring  in  the  WAPS  data  were  processed  concur¬ 
rently  according  to  the  two  models  spedfied  by  the  scenario;  i.e.,  POE  and  ITO 
measures  were  computed  for  each  of  these  engagement  situations. 


The  HIT,  high  interest  track,  broadcasting  scheme  optimizes  channel  and  computer  utilization  by  for- 
warding  only  the  tracks  required  by  the  H!T-rece1v1ng  participants.  It  is  not  bidirectional  and  uses  low  data- 
rate  radio  channels,  usually  on  HF  for  beyond  line-of-sight  radio-wave  propagation. 
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TABLE 


Simulation  parameters  used  in  the  computation  of  DREV  MBMs 
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This  is  the  general-reward  value  of  the  general-engagement  measure  (GEM),  which  is  an  early  version  of 
the  POE  (Ref.  5). 

This  is  the  spedfic-reward  value  of  the  specific-engagement  measure  (SEM),  which  is  an  early  version  of 
the  ITO  (Ref.  5), 
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3  PROCESSING  OF  OUTPUTS  FROM  SIMULATION  RUNS 


The  results  of  each  simulation  run  are  combined  in  an  output  file  where 
each  record  holds  the  results  of  the  application  of  the  two  models  selected  in  the 
scenario  to  a  single  engagement.  A  record  contains  the  following  information  on  an 
engagement: 

Engagement  profile 

1 .  engaging  ship  identification; 

2.  engaging  ship  GT  location  at  time  of  dedsion; 

3.  time  of  dedsion; 

4.  intended  target  identification  (as  reported  in  the  commander’s 
WAPS); 

5.  intended  target  location  (as  reported  or  predicted  from  the  com¬ 
mander's  WAPS); 

6.  distance  between  engaging  ship  and  target  location; 

7.  indicator  of  intersection  between  the  WUA  and  the  TUA  of  the  actual 
location  of  the  target  (according  to  the  GT); 

8.  identification  of  all  CUAs  intersecting  each  WUA  for  every  engagement 
opportunity  and  scenario. 

Engagement  outcome  assessment 

1 .  hit  probability  (indiscriminate  for  all  intersecting  CUAs); 

2.  pertinence  reward  value  (PRV); 

3.  POE  measure  of  the  engagement; 

4.  target  reward  value  (TRV); 

5.  ITO  measure  of  the  engagement. 
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Each  simulation  run  produces  one  such  file  of  raw  results;  thus  the  set  of 
simulation  runs  yielded  48  result  files. 

Each  result  file  is  processed  by  a  classification  program  that  extracts  the 
data  from  individual  records  and  groups  them  into  files  of  classes,  two  for  each  re¬ 
sult  file.  The  first  groups  the  results  according  to  the  distance  between  the  en¬ 
gaging  ship  and  the  target  location  (reported  or  extrapolated);  the  second  accord¬ 
ing  to  the  value  of  the  POE  and  ITO  measures. 

The  distance  classification  file  holds  eight  records,  one  for  each  distance 
class  considered.  Each  record  contains  the  following  information  on  a  distance 
class: 

1 .  distance  class  range  (from  100  to  900  km  in  class  vsndths  of  100  km); 

2.  class  count  (the  number  of  cases  for  which  the  distance  fell  within 
that  class  range  in  the  result  file); 

3.  number  of  intersections  (the  number  of  times  the  TUA  of  the  actual 
target  location  intersected  the  WUA,  for  all  engagements  in  that 
class); 

4.  class  rate  of  intersection; 

5.  mean  POE  measure  of  all  engagements  in  the  class  range; 

6.  mean  ITO  measure  of  all  engagements  in  the  class  range. 

The  file  that  classifies  engagement-opportunity  measures  holds  six  records, 
one  for  each  class  of  measure  value  considered.  Each  record  contains  the  following 
information  on  a  class  of  measure  values: 

1 .  class  range  of  values  (from  -2.5  to  0.5  in  class  widths  of  0.5); 

2.  class  POE  count  (the  number  of  engagement  cases  for  which  the  POE 
measure  fell  within  that  class  range  in  the  result  file); 

3.  number  of  POE  intersections  (the  number  of  times  the  TUA  of  the  ac¬ 
tual  target  location  intersected  the  WUA  for  all  engagements  whose 
POE  measure  fell  within  the  class); 
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4.  class  POE  rate  of  intersection; 

5.  mean  distance  between  engaging  ship  and  target  location  (reported  or 
extrapolated)  of  all  engagements  whose  POE  measure  fell  within  the 
class  range; 

6.  class  ITO  count  (the  number  of  engagement  cases  for  which  the  ITO 
measure  fell  within  the  class  range  in  the  result  file); 

7.  number  of  ITO  intersections  (number  of  times  the  TUA  of  the  actual 
target  location  intersected  the  WUA  for  all  engagements  whose  ITO 
measure  fell  within  the  class); 

8.  class  ITO  rate  of  intersection; 

9.  mean  distance  between  engaging  ship  and  target  location  (reported  or 
extrapolated)  of  all  engagements  whose  ITO  measure  fell  within  the 
class  range. 

The  classification  files  are  regrouped  into  spreadsheets.  One  spreadsheet  set 
regroups  the  distance  classification  files:  each  holds  the  data  of  the  four  distance 
classification  files  (one  for  each  WAPS)  according  to  one  of  the  three  scenarios 
used  and  for  one  particular  day  of  TIMSIM  ’93  data.  A  second  set  regroups  the  data 
from  the  classification  files  of  the  engagement  opportunity  measures.  The  reader 
is  referred  to  Ref.  8  for  the  detailed  results  contained  in  these  spreadsheets. 

Finally,  these  results  (and  some  supplementary  ones)  are  summarized  in  six 
spreadsheets  gathering  overall  results  for  rates  of  intersections,  POE  measures,  ITO 
measures,  time  delays,  positional  errors  including  all  cases  and  positional  errors 
with  "outliers”  removed  ("outliers”  are  defined  in  Chapter  4).  They  are  presented 
in  the  Appendix  C  of  this  document  and  discussed  in  Chapter  5. 
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4  SUPPLEMENTARY  ANALYSES  AND  RESULTS 

In  addition  to  the  analyses  performed  based  on  dedsion  models,  several 
standalone  programs  v^ere  applied  to  the  TIMSIM  '93  data  to  examine  descriptive 
statistics  about  delays  and  positional  errors.  The  results  are  gathered  into  three 
sets  of  spreadsheets,  which  are  included  in  Ref.  8  and  from  which  we  have  ex¬ 
cerpted  the  data  for  this  Chapter  and  for  Appendix  C. 

The  first  set  of  spreadsheets  looks  at  the  distribution  of  delays  in  the  WAPS 
reports,  i.e.,  the  interval  between  the  time  of  report  and  the  position  time.  There 
is  one  spreadsheet  per  day  of  TIMSIM  '93  data,  each  containing  the  results  for  the 
four  WAPS.  Allegiances  are  discriminated  and  results  are  grouped  into  five  time 
classes,  from  game  minute  2300  to  game  minute  2900  in  steps  of  120  minutes. 
Cases  are  distributed  among  these  classes  according  to  the  position  time  in  the 
WAPS  records.  The  number  of  cases  and  the  average  of  the  delays  for  each  class  is 
computed. 

The  second  set  of  spreadsheets  gathers  information  about  positional  errors 
in  the  WAPS  databases.  There  is  one  spreadsheet  for  each  day  of  TIMSIM  '93  data. 
For  each  report  in  a  WAPS,  the  mean  positional  error  is  computed  as  the  distance 
between  the  position  reported  in  the  WAPS  and  the  position  in  the  GT  at  position 
time.  These  errors  are  cumulated  over  all  reports  in  the  WAPS  and  their  mean  is 
computed  for  classes  of  distances  (distance  between  the  WAPS-reported  contact 
position  and  the  center  of  the  corresponding  group  of  blue  ships)  ranging  from  0  to 
900  km  in  steps  of  100  km,  all  according  to  the  reported  allegiance  of  the  contact. 
WAPS  reports  lacking  identifications  (class-names)  or  with  unresolved  allegiances 
are  dismissed. 

Examination  of  the  positional  error  spreadsheets  reveals  several  cases  in 
which  the  mean  error  is  totally  unreasonable.  By  investigating  the  WAPS  data 
source  we  found  that  these  instances  arose  because  of  incorrect  class-name  identi¬ 
fication  by  the  tracker/correlator  in  the  WAPS  files.  Although  there  were  very  few 
such  cases  (fewer  than  0.5%  of  the  total  number),  their  affect  was  strong  enough 
to  greatly  affect  the  average  of  the  class.  Consequently,  a  third  set  of  spread- 
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sheets  was  produced  using  the  same  computations  but  removing  data  points  whose 
positional  error  exceeded  30  km  at  source  or  sensor  time  (position  time).  We  call 
these  cases  "outliers.” 

4. 1  Discussion  of  Results  from  the  Supplementary  Analyses 

Three  summary  spreadsheets,  included  in  Appendix  C,  were  extracted  from 
these  three  sets  of  source  spreadsheets.  They  regroup  the  previous  results  for  each 
day  of  TIMSIM  '93  data  and  for  each  WAPS  without  considering  the  granularity 
among  classes  of  game  time  and  range  found  in  the  source  spreadsheets.  They  are 
discussed  in  the  next  subsection. 

4.1.1  Positional  Errors 

A  visual  inspection  of  the  positional  error  data  (including  all  cases)  at  posi¬ 
tion  time  in  the  summary  Table  of  Appendix  C  shows  that  in  several  instances  the 
error  is  excessive;  for  example,  it  is  205.18  km  for  hostile  contacts  in  SAG1  on  Day 
2  and  144.77  km  for  neutral  contacts  in  SAG2  on  Day  3.  These  figures  are  averages, 
indicating  even  greater  discrepancies  for  individual  reports.  Although  this  situation 
occurred  mostly  for  hostile  and  neutral  contacts,  it  affects  the  overall  error  meas¬ 
ures  significantly.  Errors  of  such  magnitude  are  completely  unexpected:  since  we 
are  looking  at  the  positional  error  at  position  time,  we  would  expect  errors  of  the 
order  of  magnitude  of  the  sensor  positional  accuracy. 

As  was  the  case  in  the  previous  section,  tracking  down  the  sources  of  these 
discrepandes  revealed  a  few  individual  instances  were  the  error  was  so  high  that  it 
biased  the  average  of  the  whole  class.  These  errors  were  introduced  by  incorrect 
class-name  identification  from  the  tracker/correlator.  These  particular  cases  can 
be  viewed  as  aberrations  whose  removal  does  not  affect  the  statistical  significance 
of  the  results,  because  they  are  few  in  number  compared  to  the  sample  sizes.  On 
the  other  hand,  not  removing  them  would  lead  to  false  conclusions  being  drawn 
from  the  results.  In  the  analysis  of  future  exerdses  it  would  be  desirable  to  iden- 
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tify  such  discrepandes^s,  count  them,  to  report  their  causes  and  to  remove 
them  from  the  positional  accuracy  estimators,  since  they  properly  belong  to  other 
measures,  such  as  coherency  or  human  error  rate  (see  RIMPAC  ’94  and  '96,  the 
ground-truth  assessment). 

Filtering  out  these  cases  by  setting  the  maximum  allowable  positional  error 
to  30  km  yields  the  results  given  in  the  summary  Table  in  Appendix  C:  positional 
errors  with  outliers  removed.  More  realistic  figures  are  seen  at  once  in  the  sum¬ 
mary  results  presented  in  Table  IV.  In  fact,  the  error  values  indicate  very  good  sen¬ 
sor  accuracy.  For  each  day,  the  hostile  category  has  a  higher  positional  error  than 
the  friendly  and  neutral  categories.  This  seems  reasonable  since  it  is  expected  that 
the  enemy  would  be  more  difficult  to  track  predsely  than  friendly  or  neutral  units 
because  of  evasive  maneuvers  on  their  part  or  the  use  of  countermeasures  against 
sensor  operations.  Friendly  units  report  their  positions  with  higher  accuracy  than 
provided  by  some  sensors,  while  neutral  units  are  more  likely  to  follow  steady 
courses,  thus  favoring  better  predsion  in  estimates  of  their  positions. 

The  overall  figures  shown  in  Table  IV  do  not  indicate  much  difference  from 
day  to  day  or  between  WAPS  views,  with  the  exception  of  Day  1  where  the  FOTC, 
SAG1  and  SAG2  have  significantly  lower  errors  than  on  the  other  days  and  for  the 
TWCS,  which  shows  a  lower  error  than  for  the  three  JMCIS  nodes  for  every  day  but 
Day  1. 


Removing  the  outliers  reduced  the  apparent  mean  positional  error  at  posi¬ 
tion  time  from  24  km  to  2  km,  ten  times  lower. 


Tracing  the  source  of  such  errors  may  help  identifying  problems  with  wide-area  tracker/correlator  algo- 
rithms  and  procedures.  Finding  a  way  to  identify  them  in  real-time  would  reduce  the  annount  of  misleading 
information  provided  to  decision  makers  and  consequently  might  avoid  possible  blue-on-blue  engagements. 
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TABLE  IV 

Overall  positional  errors  without  the  outliers  versus  (::)  the  unfiltered 
data  (excerpt  from  the  summary  tables  of  Appendix  C) 


4.1.2  Delays 

Delays  were  measured  by  taking  the  difference  between  the  report  time  and 
the  position  time  in  the  contact  information  reports.  The  summary  table  for  delays 
given  in  Appendix  C  shows  some  values  that  are  unexpectedly  high.  This  is  espe- 
dally  so  on  Day  1:  e.g.,  averages  of  46.63,  47.74  and  54.75  minutes  for  hostile, 
friendly  and  neutral  contacts  in  TWCS  and  an  average  of  50.15  minutes  for  hostile 
contacts  in  SAG1.  Many  operational  problems  were  encountered  on  Day  1  of  TIMSIM 
'93,  so  it  is  reasonable  to  assume  that  systems  and  processes  were  not  properly 
tuned  that  day.  There  is  also  a  possibility  of  an  effect  similar  to  the  "outliers”  ef¬ 
fect  on  positional  errors:  in  the  case  of  delays,  outliers  might  be  reports  on  stalled 
tracks:  successive  reports  sent  on  a  contact  without  updating  the  position  data. 
Just  a  few  of  these  cases  lasting  for  hours  would  greatly  affect  the  average  delay 
measure.  Although  such  instances  were  identified  in  the  analyses,  they  have  not 
yet  been  investigated  systematically  to  determine  their  number,  source  or  average 
duration.  It  is  expected  that  the  removal  of  a  few  cases  with  very  high  delays  due 
to  stalled  tracks  would  significantly  lower  the  averaged  results. 
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During  RIMPAC  '94  and  '96,  maximum  delay  values  were  observed  much  in 
excess  of  those  indicated  above.  For  example,  Ref.  10  indicates  that  track  reports 
for  opposed-force  FOTCs  were  as  much  as  90  minutes  old  for  7%  (one  side)  and  26% 
(for  its  opponent)  of  their  hostile  tracks,  for  2%  and  1%  of  their  friendly  ones  and 
for  2%  and  14%  of  the  other  tracks  in  the  blue  and  orange  force  FOTC  databases, 
respectively.  These  large  delays  were  mainly  attributable  to  the  lack  of  appropri¬ 
ate  automated  support  to  the  surface-picture  managers  by  systems  like  JMCIS  or 
the  GCCS  using  the  FOTC  concept.  These  figures  are  noted  to  indicate  that  one 
cannot  simply  remove  unreasonable  values  without  tradng  their  causes.  But  when 
these  anomalous  values  are  few  in  number,  fewer  than  2%  of  the  number  of  re¬ 
ports,  and  their  effect  on  the  overall  statistics  is  significant,  it  is  advisable  to  ex¬ 
clude  them. 

This  aspect  of  the  analysis  leads  naturally  to  a  recommendation  for  future 
systems.  There  is  a  strong  requirement  for  better  aids  to  track  managers  in  JMCIS- 
or  GCCS-like  systems  used  to  support  the  FOTC  functions  or  shore  fusion  centers 
like  MCOIN:  they  and  their  users  must  be  alerted  to  high-interest  tracks  that  are 
overdue  for  updating.  Track  symbology  must  indicate  data  aging  beyond  an  agreed- 
upon  threshold  for  this  type  of  contact.  Then,  if  necessary,  surveillance  assets  can 
be  deployed. 

As  Table  V  indicates,  with  the  exception  of  Day  1  the  overall  delays  for 
TWCS  are  significantly  lower  than  for  the  three  JMCIS  nodes  and  the  FOTC  shows 
delays  always  lower  than  those  for  SAG1  and  SAG2.  The  difference  in  delays  be¬ 
tween  SAG1  and  FOTC  and  the  difference  between  SAG2  and  FOTC  are  good  indica¬ 
tors  of  the  dissemination  time  from  the  FOTC  to  these  two  JMCIS  nodes.  These  dif¬ 
ferences  are  almost  identical,  delays  to  SAG  2  being  slightly  larger  than  those  to 
SAG  1,  and  we  conclude  that  for  the  volume  of  data  transferred  the  low  communi¬ 
cation  channel  capacity  to  SAG  2  had  less  effect  than  the  FOTC  procedure  that  im¬ 
poses  a  15-minute  wait  before  forwarding  updated  pictures  to  partidpants. 


UNCUSSIFIED 

32 


TABLE  V 

Overall  delays  (excerpt  from  the  summary  tables  of  Appendix  C) 


i:  delays  . 


mcs  i  FOTC;  SAG1  l$AGZ 
51,07  37.67  ipP  • 

12.95  '/t9,25  '  26*82  ;!  34,54 

. v-rr . 

13.46  ||B|  28,94  ®B| 
f/  0^4'^'  '  :  11.62  ;'21.5$i  28.74  ^27,56, 


f''T:T~  . . . 


ov^ell 


5  22,27  ;  19,50  30.54  30,57 


Delays  and  positional  errors  should  be  strongly  correlated  since,  in  a  dy¬ 
namic  environment,  aging  of  positional  data  can  be  expected  to  induce  errors  in 
position  as  a  function  of  time.  This  correlation  cannot  be  established  here  because 
the  previous  results  on  positional  errors  were  all  gathered  at  position  time.  To  es¬ 
tablish  the  correlation  with  the  delays  results  of  Table  V  we  would  need  to  look  at 
the  positional  errors  at  report  time  and  at  the  difference  between  that  error  and 
the  positional  error  at  position  time;  that  is,  to  look  simultaneously  at  the  varia¬ 
tion  in  positional  errors  from  position  time  to  report  time  as  the  delay  increases. 
We  have  not  yet  quantified  this  correlation,  but  all  the  necessary  data  is  at  hand 
and  we  hope  to  report  on  it  in  future  work.  The  phenomenon  is  outside  the  scope 
of  this  document  and  results  on  such  analyses  can  be  found  elsewhere,  e.g.. 
Refs.  10,  16. 


The  question  of  removing  cases  of  extreme  values  for  delays  (as  well  as  for 
positional  errors)  has  both  statistical  and  logical  foundations.  First,  it  is  not  un¬ 
common  in  data  analysis  to  filter  eccentric  values  or  aberrations.  If  they  are  due  to 
an  incorrect,  erroneous  or  ill-conditioned  behavior  of  the  system  being  studied  and 
one  is  not  spedfically  investigating  that  behavior,  it  makes  sense  to  eliminate  the 
resulting  data  gathered,  as  long  as  the  number  of  such  occurrences  is  not  propor¬ 
tionally  too  large.  Second,  one  objective  of  our  analysis  is  to  assess  the  impact  of 
system  changes  on  overall  picture  quality  and  on  mission  effectiveness.  Such 
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changes  might  be  related  to  performance  (more  and  better  sensors,  increased 
bandwidth  and  data  transmission  rates)  or  effectiveness  (data  fusion,  integration, 
dissemination).  We  want  to  be  able  to  compare  these  changes  in  an  otherwise  sta¬ 
ble  environment,  one  in  which  unrealistic,  exceptional  or  ill-conditioned  behavior 
of  the  systems  from  one  setup  to  another  does  not  compromise  the  soundness  of 
inferences  made.  The  analysis  results  will  lead  to  most  significant  conclusions 
when  we  consider  systems  operating  under  "normal”  conditions.  This  does  not 
mean  that  we  are  restricted  to  steady-state  systems,  but  that  the  amount  of  vari¬ 
ability  be  realistic  and  representative  of  usual  conditions  of  operations.  Excep¬ 
tional  events  due  to  abnormal  or  exceptional  conditions  must  be  dealt  with  sepa¬ 
rately  and  are  part  of  other  types  of  measures. 
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5  INTERPRETATION  OF  THE  MODEL-BASED  MEASURES 

The  actual  values  of  the  POE  and  ITO  measures  range  from  -2.5  to  0.5. 
These  figures  do  not  convey  an  absolute  but  rather  a  relative  appredation  of  en¬ 
gagement  outcomes.  Running  the  same  analyses  with  different  values  for  the  COF 
and  the  allegiance  factors  would  yield  a  different  range  of  values. 

With  the  set  of  values  actually  used  for  these  parameters,  the  interpretation 
of  the  measures  within  the  range  is  quite  simple.  The  upper  bound  of  0.5  indicates 
an  optimal  engagement  outcome:  a  sure  hit  to  an  hostile  ship  (POE  measure)  or  to 
the  intended  hostile  target  (ITO  measure).  The  lower  bound  of  -2.5  is  the  worst 
outcome  possible:  a  sure  hit  to  a  friendly  or  a  neutral  ship  (we  used  the  same  alle¬ 
giance  factor  value  for  these  two  categories). 

Between  these  two  extremes,  we  propose  the  following  steps  for  interpret¬ 
ing  the  intermediate  values.  We  assume  that  this  scale  applies  to  the  aggregated 
measures  (overall)  found  in  the  summary  tables  of  Appendix  C,  thus  to  the  average 
of  the  measures  on  a  large  number  of  engagement  situations  (several  thousands, 
Refs.  5,  8).  Table  VI  shows  a  possible  scheme. 

5.1  Characterizations  of  the  POE  and  ITO  Persona 

The  characterization  of  the  POE  and  ITO  measures  presented  above  is  a  first 
attempt  to  provide  a  simple  and  meaningful  interpretation  of  the  model  measures 
in  terms  of  mission  effectiveness  or  engagement  dedsion  productivity.  We  have 
applied  descriptive  labels  to  equally  spaced  levels  in  the  numerical  range  of  possi¬ 
ble  values  for  the  measures  in  order  to  give  the  measures  a  more  natural  interpre¬ 
tation.  It  can  be  argued  whether  the  number  of  levels  defined  over  the  whole  in¬ 
terval  (7)  is  too  large  or  too  small.  The  actual  terms  used  for  labels  are  also  moot. 
However  it  seems  reasonable  that  the  label  term  used  for  the  lowest  numerical 
value  of  the  measures  should  indicate  a  very  unfavorable/unproductive  engage¬ 
ment  outcome  and  that  the  label  term  used  for  the  highest  possible  value  of  the 
measures  should  mean  a  very  favorable/productive  one.  Between  these  two  ex¬ 
tremes  the  label  terms  should  indicate  an  increasingly  favorable  outcome. 
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TABLE  VI 

Proposed  mapping  between  a  MBM  computed  value  and  its  interpretation, 

a  cardinal  or  label 


value 

range 

description  and  label 

0.5 

most  productive  situation,  enemy  ships  are  always  hit,  this  level  will  be  labeled 
"best”  hereafter 

JO.O,  0.5] 

highly  productive 

0.0 

productive  situation;  a  typical  case  yielding  this  ITO  value  would  be  to  hit  a  hos¬ 
tile  ship  other  than  the  intended  one.  "Good”  will  be  used  as  a  label  for  this 
level  of  success 

J-0.5,  0.0] 

marginally  productive 

-0.5 

"marginal”  level;  anything  below  this  point  is  counterproductive.  A  typical  case 
is  when  there  is  no  contact  whose  CUA  intersects  the  WUA  and  the  weapon  is 
fired  uselessly 

]-1.0,  -0.5] 

marginally  counterproductive 

-1.0 

counterproductive  cases  outweigh  productive  ones:  e.g.,  only  three  ships  out  of 
four  intercepted  are  orange;  label  "counterproductive”  due  to  the  COF 

]-1.5,-1.0] 

very  counterproductive 

-1.5 

"extremely  counterproductive,”  e.g.,  only  two  orange  ships  out  of  three  ships 
intercepted 

]-2.0,  -1.5] 

exceedingly  counterproductive 

-2.0 

undesirable  situation  in  which  the  blue  force  destroys  as  much  of  its  own  or  neu¬ 
tral  ships  as  it  destroys  the  orange;  label  "quasi  fratricide” 

[-2.5,  -2.0] 

fratricide 

-2.5 

most  undesirable  situation  in  which  blue  force  destroys  only  neutral/friendly 
ships  rather  than  hostile  ones;  this  level  is  labeled  "worst”  or  "total  fratricide” 

Using  equally  spaced  levels  within  the  interval  range  more  or  less  assumes  a 
degree  of  linearity  in  the  increase  of  the  "goodness”  of  the  outcome.  This  may  not 
necessarily  be  the  case.  For  example,  it  is  possible  that  a  small  increase  in  the 
numerical  values  at  the  high  end  of  the  scale  results  in  a  more  significant  increase 
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of  productivity  or  effectiveness  than  the  same  increase  in  the  middle  or  lower  part 
of  the  scale. 

Adjustments  can  be  made  by  selecting  unequal  subinterval  lengths  between 
adjacent  levels  or  by  using  a  sequence  of  label  terms  whose  connotation  does  not 
vary  linearly  from  one  label  term  to  the  next.  The  latter  means  is  very  question¬ 
able,  since  it  relies  highly  on  the  subjective  interpretation  of  the  label  terms  and  it 
would  be  very  difficult  to  choose  a  set  of  terms  that  would  consistently  represent 
an  appropriate  increase  in  productivity  from  one  step  to  the  next.  The  former 
means  appears  more  practicable;  i.e.,  use  unequal  subinterval  lengths  between 
adjacent  levels.  We  propose  to  use  a  very  simple  set  of  label  terms,  one  containing 
from  four  to  eight  very  basic  terms  possibly  qualified  by  intensification  adjectives. 
They  should  be  chosen  so  that  when  they  are  ordered,  they  convey  a  more  or  less 
constant  increase  in  the  intensity  of  their  connotation.  For  example,  "worst,” 
"bad,"  "insignificant,”  "good,”  "best.”  These  would  then  be  applied  to  unequally 
spaced  levels  over  the  whole  interval  of  values  according  to  the  actual  correspon¬ 
dence  or  distribution  of  the  measure. 

The  problem  of  selecting  the  lengths  of  the  subintervals  or,  equivalently, 
the  unequally  spaced  level  points  is  the  most  difficult  part.  In  the  absence  of  cu¬ 
mulated  results  from  an  insightful  set  of  exerdses,  one  may  apply  a  mapping  based 
on  a  threshold,  with  all  values  above  the  threshold  mapped  to  a  value  from  0.0%  to 
100%.  At  the  threshold  the  value  is  0%.  Values  below  the  threshold  are  negative 
percentages,  follovring  the  scaling  imposed  by  the  positive  values.  Furthermore, 
since  we  are  interested  in  productive  systems  that  are  capable  of  improving  mis¬ 
sion  success  rates,  the  focus  must  be  on  the  positive  segment  of  this  mapping,  i.e., 
from  0%  to  100%,  which  might  correspond  to  cardinal  values  marginal  to  best. 

5.1.1  Mapping  between  MBM  Scales 

A  tentative  mapping  between  these  representations  and  the  scales  for  the 
present  MBMs  and  their  former  versions  is  illustrated  in  Fig.  7.  We  assume  that  the 
scales  are  imposed  by  the  new  and  old  DREV  MBM  definitions  and  by  the  parame¬ 
ters  spedfied  above.  By  setting  the  threshold  at  -0.5  for  the  new  MBMs  we  can 
force  their  percentage  values  to  approach  closely  those  of  the  previous  MBM  ver- 
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FIGURE  7  -  Scales  defined  by  DREV’s  MBMs  for  both  versions 

Sion.  This  mapping  is  valid  only  when  the  rate  of  successful  engagements  is  much 
larger  than  that  of  failures.  It  also  requires  the  assumption  of  the  parameters  indi¬ 
cated  previously  for  the  COF,  CUA,  WUA  and  rewarding  values  per  allegiance:  the 
current  version  with  a  COF  of  -0.5  and  a  maximum  penalty  of  -2;  the  previous  ver¬ 
sion  with  no  COF  and  a  maximum  penalty  of  -1 . 

The  raw  results  from  the  POE  and  the  ITO  that  include  any  significant  COF 
are  much  lower  in  value  than  those  obtained  with  the  measures  defined  for  the 
previous  MBMs  (which  do  not  include  a  COF  component).  However,  the  new  meas¬ 
ures  deliver  relative  results  that  are  almost  identical  to  those  given  by  the  early 
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version  of  the  MBMs.  The  new  version  (including  COF  and  other  resource  contingen¬ 
cies)  described  in  this  document  offers  the  advantage  of  being  closer  to  a  realistic 
scale,  although  empirical  calibration  is  still  required,  particularly  to  provide  spe- 
dfic  COF  values  for  each  combination  of  weapon  and  target,  based  on  tactical  and 
economic  impact. 

5.1.2  Statistical  Characterization 

As  experience  is  gained  with  the  models  by  applying  them  to  larger  sets  of 
controlled  experiments,  a  statistical  basis  of  inference  for  the  behavior  of  the 
overall  measures  should  emerge,  independent  of  the  COF  and  allegiance  factors. 
This  would  allow  a  more  refined  stratification  of  assessment  where  the  intervals  of 
values  for  the  overall  measures  are  not  necessarily  the  same  for  each  mission  ef¬ 
fectiveness  assessment  label,  e.g.,  extremely  effective  or  productive,  very  effec¬ 
tive,  effective,  poor,  and  so  on.  The  statistics  should  be  separate  for  each  scenario 
(delay,  no  delay,  delay  with  prediction),  for  each  measure  (POE  and  ITO)  and  for 
each  WAPS  (TWCS,  FOTC  and  others),  since  samples  should  be  drawn  from  homo¬ 
geneous  populations.  Though  the  range  of  possible  values  for  a  given  statistic  may 
change  according  to  values  set  for  the  COF  and  the  allegiance  factors,  the  general 
shape  of  the  distribution  is  likely  to  be  the  same  over  several  experimental  sam¬ 
ples. 


For  example,  Pmd  (the  probability  variable  POE  for  the  TWCS  with  no  delay) 
may  be  defined  as  a  stochastic  variable  representing  overall  POE  measures  sam¬ 
pled  on  a  set  of  experiments  (or  subsets  of  engagements  from  a  set  of  experi¬ 
ments)  using  "no  delay”  scenarios  and  TWCS  information  systems.  Statistical  vari¬ 
ables  defined  under  these  conditions  (the  same  type  of  information  system  and  the 
same  type  of  scenario)  should  exhibit  the  typical  distribution.  That  distribution 
may  not  be  the  same  for  variables  defined  over  different  information  systems  and 
different  scenarios  or  it  may  be  a  similar  function  but  vrith  different  characteristic 
parameters,  e.g.,  mean  and  variance.  Possibly  the  "no  delay”  and  "delay  vrithout 
prediction”  populations  (for  the  same  type  of  information  system)  could  be 
merged  into  a  single  population  of  experiments  since  the  "no  delay”  scenario  du¬ 
plicates  the  lower  extreme  values  of  the  "delay  without  prediction”  scenario. 
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In  order  to  find  the  distributions  of  typical  measures  that  are  characterized 
by  a  set  of  conditions  (e.g.,  the  same  type  of  information  system  and  scenario), 
exploratory  empirical  distributions  are  generated  by  selecting  susceptible  sets  of 
conditions  for  which  we  expect  to  observe  differences.  In  such  cases  the  experi¬ 
ence  of  the  analysts  must  determine  the  course  of  action  to  be  taken,  otherwise 
one  may  focus  on  non-significant  variations  of  the  data  for  a  particular  study. 

5.2  Results 

5.2.1  Intersection  Rates 

In  each  series  of  results  from  the  application  of  the  six  models  described  in 
Chapter  2  the  "intersection  rates”  of  CUAs  vrith  WUAs  is  a  primary  result  that  is 
used  in  MBM  computation.  Intersection  rates  without  delay,  with  delay  and  no  pre¬ 
diction,  and  with  delay  and  prediction  have  a  strong  correlation  vrith  their  corre¬ 
sponding  MBMs.  These  intersection  rates  express  the  scenario  prerequisite  condi¬ 
tions  for  an  engagement  situation  for  the  six  MBMs. 

Table  VII  lists  the  overall  intersection  rates  (see  Appendix  C).  A  comparative 
bar  chart  illustrates  these  values  in  Fig.  8.  The  intersection  rates  represent  the 
proportion  of  engagement  situations  where  the  CUA  of  the  hostile  target  inter¬ 
sected  the  WUA  at  the  time  of  decision.  These  are  important  indicators  directly 
related  to  the  effectiveness  of  an  engagement  decision.  Indeed,  notwithstanding 
the  possibility  of  other  contact  CUAs  intersecting  the  WUA,  a  higher  intersection 
rate  favors  a  successful  engagement  outcome.  The  chart  shows  a  consistent  degra¬ 
dation  of  intersection  rates  from  the  ideal  model  involving  no  delay,  through  delay 
applied  without  prediction,  to  delay  with  dead-reckoning.  Unsurprisingly,  all  the 
WANTAP  systems  considered  are  affected  by  this  degradation  due  to  information 
aging,  so  it  is  reasonable  to  assume  this  behavior  to  be  independent  of  the  par¬ 
ticular  information  system  considered  (but  not  independent  of  its  timeliness).  Us¬ 
ing  a  different  predictive  scheme  might  improve  the  intersection  rates  for  the  de- 
lay-with-prediction  class  if  such  a  scheme  were  to  mimic  more  closely  the  ex¬ 
pected  maneuvering  of  hostile  ships  after  their  detection.  The  scenarios  and  inter¬ 
sections  are  explained  with  the  definitions  and  related  illustrations  (Figs.  1-6)  pre¬ 
sented  previously. 
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TABLE  VII 

Intersection  rates  (%) 


;  TWCS 

fOTC  SA61 

smi 

. 

■|  9t.1 

' 

84.2 

Bl.l 

62.3 

.^snV'- 

\m  45.7 

::  Xv:;: 

;,46.3  ' 

1  47.5 

..  ^ 

5 

[454' 

f  33.8 

■  36  J 

FIGURE  8  -  Chart  of  intersection  rates 


The  intersection  rates  for  the  data  without  delay  from  the  sensors  indicate 
the  ability  of  the  sensor  suite  feeding  a  WAPS  to  estimate  the  position  and  alle¬ 
giance  of  contacts  accurately.  We  use  this  for  our  sensor  baseline.  Nominally,  from 
the  point  of  view  of  improving  the  reported  tracks  scrutinized  by  this  measure,  the 
maximum  achievable  theoretical  improvement  at  the  TWCS  is  9%"'’.  This  limit  dis¬ 
regards  the  limitations  due  to  physics  or  to  scenario  constraints,  so  for  the  hostile- 
ship  picture  the  maximum  feasible  sensor  improvement  must  be  much  less  than 
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We  found  8%  in  a  previous  analysis  of  the  raw  data  from  the  same  experiment  (Refs.  5,  12). 


UNCLASSIFIED 

41 


TABLE  VIII 

Overall  POE  measure 


FIGURE  9  -  POE  engagement  action  productivity  (pertinence) 

9%.  Since  our  measure,  this  intersection  rate,  is  invariant  to  completeness,  it  is  not 
affected  by  sensor  deployment  but  by  the  ability  of  the  sensors  to  provide  correct 
allegiance  and  position. 

5.2.2  Pertinence-of-engagement  Measure  Results 

Table  VIII  lists  the  overall  POE  measure  results  (see  Appendix  C).  These  re¬ 
sults  are  converted  and  charted  in  Fig.  9  by  applying  the  previous  interpretative 
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labels  to  the  actual  numerical  values.  The  same  typical  behavior  appears,  i.e.,  a 
degradation  in  productivity  (mission  effectiveness)  as  delay  is  introduced  in  the 
model;  more  so  if  dead-reckoning  prediction  is  used.  Though  the  scales  used  are 
not  the  same  for  the  POE  measure  and  the  intersection  rate,  there  is  even  more 
discrepancy  here  between  the  delay  and  no-delay  models.  The  intersection  rate 
does  not  account  for  the  distance  between  a  contact  whose  CUA  intersects  the 
WUA  and  the  center  of  the  WUA.  The  POE  measure  accounts  for  that  distance  by 
assigning  a  lower  probability  of  being  hit  as  the  distance  increases.  Since  the  intro¬ 
duction  of  delays  in  the  data  increases  positional  errors,  a  greater  discrepancy  is 
expected  for  the  POE  measure.  In  addition,  the  intersection  rate  is  not  sensitive  to 
the  presence  of  other  contacts  whose  CUA  might  intersect  the  WUA,  but  the  POE 
measure  is  very  responsive  to  the  presence  of  other  contact  CUAs  in  the  vidnity  of 
the  WUA  and  to  differences  in  allegiance:  a  possible  friendly  and/or  neutral  ship  in 
the  WUA  greatly  decreases  the  POE  measure. 

5.2.3  Intended-target-opportunity  Measure  Results 

Table  IX  lists  the  overall  ITO  measure  results  (see  Appendix  C).  The  results 
are  converted  and  presented  vrith  the  proposed  labels  in  the  chart  of  Fig.  10  in  the 
same  manner  as  for  the  POE  measure.  As  one  would  expect,  the  degradation  pat¬ 
tern  is  similar  to  that  observed  for  the  POE  measure.  The  difference  lies  in  slightly 
smaller  values  for  all  groups  and  models.  This  small  difference  is  caused  by  the 
lower  allegiance  factors  attributed  to  hostile  ships  other  than  the  intended  target 
in  computing  the  ITO  measure.  However,  the  same  discrimination  among  ship  alle¬ 
giances  in  the  vidnity  of  the  WUA  applies  here,  further  emphasizing  the  gain  in 
productivity  when  there  is  no  delay. 
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FIGURE  10  -  ITO  engagement  action  prociuctivity  (intention) 


5.3  Quantitative  Comparison  between  the  MBM  Versions 

Table  X  lists  the  overall  ITO  measure  results  compared  to  the  SEM  (its  previ¬ 
ous  version),  in  percentages.  The  differences  (A  =  SEM  -  ITO)  are  presented  in 
Fig.  11. 
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TABLE  X 

Overall  ITO  measure  versus  the  SEM  measure 
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FIGURE  11  -  Difference  between  the  %  of  the  SEM  and  the  ITO  measures 


As  one  would  expect,  both  measures  offer  similar  figures  of  merit,  although 
the  new  version  of  MBMs  tends  to  deliver  lower  numerical  values  for  most  of  the 
cases  except^o  those  at  no  delay  for  the  TWCS  and  the  FOTC.  The  slightly  smaller 
values  of  the  ITO  (maximum  of  -5%)  can  be  accounted  for  by  the  larger  allegiance- 
factor  penalties  assigned  in  the  ITO  measure,  as  compared  to  the  SEM  measure 
(see  Table  III:  the  SRVs  are  -0.5  and  -1.0,  versus  -2.0  for  the  ITO).  Nevertheless, 
the  metric  was  preserved  and  conclusions  drawn  from  the  data  with  the  previous 
version  of  the  MBMs  are  essentially  the  same. 


We  suspect  that  the  removal  of  a  few  outlier  cases  in  the  new  input  data  sets  used  for  the  POE  and  ITO 
tests  may  explain  this  discrepancy,  especially  when  we  approach  higher  mission  success  rates. 
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5.4  A  Measure  of  Completeness 

For  a  given  geographical  area  and  period  of  time  and  a  spedfic  WAPS,  a  fair 
measure  of  completeness  is  how  many  and  how  often  GT  tracks  have  been  located 
and  identified  correctly.  To  establish  this  for  a  common  operating  picture  (COP) 
shared  among  dedsion  makers  we  have  to  assess  all  replicates  of  the  tentatively 
shared  database  (each  located  in  a  given  WAPS  of  a  battle  force),  then  count  only 
the  correctly  reported  contacts  that  are  common  to  all  the  replicates.  The  com¬ 
mon  part  is  the  completeness  of  the  shared  COP.  The  union  or  perfect  combination 
of  the  results  across  all  the  replicates  (the  best  WAPS  picture)  is  the  completeness 
of  the  achievable  COP,  given  no  loss  in  information  management  and  exchange. 
Differences  in  completeness  indicate  the  lack  of  coherency  among  the  replicates. 

Completeness  for  one  WAPS  assumes  the  following:  there  are  several  data¬ 
bases  to  be  sampled  and  a  GT  file  is  reconstructed  from  a  coherent  combination  of 
all  available  sources  (GPS-recorded  position  reports,  all  the  information  databases 
including  opponent  databases  in  military  exerdses  and  the  stimulating  files  used  in 
simulations).  For  all  surface  contacts  in  the  geographical  area  and  the  time  period 
spedfied  for  the  analysis,  the  GT  is  sampled  at  discrete  times  (sampling  every  15 
minutes  is  suffident  and  practical  in  most  scenarios)2i.  Each  GT  point  contains  the 
time,  the  identification  of  the  contact  and  its  allegiance.  The  completeness  meas¬ 
ure  algorithm  is  then  as  follows: 

1.  Find  the  corresponding  contact  within  a  given  CUA  for  each  GT  point  in  the  WAPS 
database  under  test. 

2.  Count  one  (1)  for  a  correctly  reported  contact  at  that  time.  If  subsequent  GT 
points  of  the  same  track  refer  to  the  same  WAPS  report  and  it  is  still  within  the 
CUA  with  the  correct  identification  continue  to  count  additional  ones  (n  •  1).  If  no 
correct  reports  corresponding  to  the  GT  point  are  found,  count  nothing  (0). 


Instead  of  using  the  elapsed  time  to  sample  the  GT  file,  one  can  use  a  thresholding  technique  (draw  a  new 
sample  when  changes  are  beyond  a  threshold  for  a  given  track  parameter).  Ref.  3.  An  evaluation  of  com¬ 
pleteness  based  on  a  threshold  technique  is  expected  to  be  more  accurate  and  fair  than  one  based  on  delays 
only.  It  is  a  kind  of  nx>del-based  measure  for  completeness  and  it  can  be  extended  to  support  the  optimiza¬ 
tion  of  surveillance-asset  deployment. 
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3.  Cumulate  the  number  of  GT  points  used  and  the  total  occurrence  of  correctly  re¬ 
ported  contacts  in  the  WAPS  database. 

4.  Compute  the  ratio  of  the  occurrence  of  correct  reports  to  the  total  number  of  GT 
points  and  report  it  as  the  measure  of  completeness  of  the  WAPS  database  for  that 
period  and  geographical  area. 

This  measure  of  completeness  of  a  tactical  picture  assesses  several  aspects 
of  the  systems,  scenarios,  dedsions  taken,  resource  management,  sensor  perform¬ 
ance  and  of  surveillance-asset  deployment.  It  is  significantly  different  from  the  in¬ 
tersection  rates  reported  for  the  MBMs.  For  the  latter  rates  we  count  the  number 
of  times  each  track  update  of  a  WAPS  intersects  the  GT,  then  analyze  only  the 
tracks  in  the  commander’s  database.  For  the  completeness  measure  we  count  the 
number  of  times  each  track  update  of  the  GT  sampled  every  15  minutes  intersects 
the  correct  WAPS  track  report. 

The  lack  of  correct  track  reports  on  some  contacts  in  the  best  WAPS  picture 
depends  strongly  on  surveillance-asset  deployment  and  sensor  performance.  At 
sensor  time,  a  lack  of  correlation  between  WAPS  track  reports  and  GT  reports  can¬ 
not  depend  strongly  on  the  deployment  of  surveillance  assets  but  rather  on  sensor 
performance^^.  At  report  time  at  the  compilation  node,  it  depends  mainly  on  in¬ 
formation  management.  At  remote  nodes,  it  depends  on  both  information  man¬ 
agement  and  communications.  Consequently,  MBMs  and  intersection  rates  do  not 
depend  strongly  on  surveillance-asset  deployment,  while  the  proposed  measure  of 
completeness  does.  This  is  illustrated  and  supported  by  the  following  results  ex¬ 
cerpted  from  previous  analyses  on  the  same  exercise  data  (Table  XI  and  Fig.  12). 


The  data  transfer  rate  from  the  sensor  to  the  compilation  node  is  assumed  to  have  little  effect  compared 
to  other  processes,  as  observed  during  certain  exerdses,  Refs.  3,  9,  10. 
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TABLE  XI 

GT  versus  WAPS  intersection  rates 


TWCS  FOTC  SAG1  SAG2 


FIGURE  12  -  GT  and  WAPS  intersection  rates  for  all  tracks  (any  allegiance)  and  hos¬ 
tile  tracks,  respectively 

It  is  worth  noting  that  the  WAPS  surface  hostile  picture  is  best  for  the 
TWCS,  as  it  should  be  for  an  over-the-horizon  weapon-control  station.  Unfortu¬ 
nately,  due  to  procedures,  data  management  (a  man-intensive  activity)  and  other 
time-consuming  activities,  the  quality  of  the  picture  is  marginal  by  the  time  the 
data  is  made  available  to  dedsion  makers,  as  shown  in  Figs  8-10.  Other  information 
sources  and  mechanisms  are  used  to  refine  the  OTH-T. 

Figure  12  shows  that  in  assessing  system  performance  and  mission  effective¬ 
ness  one  may  need  to  dissodate  the  effect  of  surveillance-asset  exploitation  (GTs) 
from  the  effect  of  exploiting  the  available  information  (WAPS). 
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From  the  information  of  this  section,  we  conclude  that  if  one  wants  to  as¬ 
sess  MOPs  or  MOEs  independently  of  surveillance-asset  deployment  then  measures 
must  be  modified  to  be  similar  to  the  MBMs,  for  example  by  selecting  only  the 
concerned  WAPS  data  and  using  the  GT  to  assess  their  quality,  rather  than  starting 
from  the  GT  data  and  searching  for  counterparts  in  the  WAPS  data,  or  by  increas¬ 
ing  the  measure  sensitivity  by  rejecting  data  that  interferes  v^nth  the  phenomenon 
under  investigation.  The  technique  is  equivalent  to  using  appropriate  band-pass 
filtering  and  processing  to  get  a  signal  out  of  noise. 

5.5  A  First-order  Approximation  of  the  POE  or  the  ITO  Measures 

Extracting  the  current  POE  or  the  ITO  measures  is  compute-intensive,  re¬ 
quiring  the  computation  of  the  bivariate  joint  probabilities  resulting  from  the  in¬ 
tersection  of  the  CUAs  and  WUA.  These  probabilities  can  be  computed  by  using  ei¬ 
ther  brute-force  Monte  Carlo  or  numerical-integration  techniques.  For  each  inter¬ 
section  we  use  a  thousand  point  semi-Monte-Carlo  technique,  described  in  Ref.  5. 
This  is  the  compute-intensive  aspect  of  the  DREV  MBMs.  For  data  files  too  large  for 
the  fast  memory  of  the  computer,  which  therefore  require  multiple  accesses  to 
disk,  the  proposed  approximation  may  not  provide  as  great  a  performance  im¬ 
provement  as  is  indicated  below,  but  the  speedup  will  still  be  impressive. 

The  first-order  approximation  algorithm  is  as  follows: 

1.  Find  the  GT  track  sample  at  decision  time  (position  time  and  report 
time,  or  a  later  time  for  decision  making)  that  is  the  closest  to  the 
track  report  update  that  triggered  the  measure.  Then  compute  the 
apparent  miss-distance23  or  targeting  positional  error. 


This  is  the  pointing  error  and  not  the  expected  miss  distance,  which  requires  a  knowledge  of  the  detailed 
engagenf)ent  scenario,  the  missile  characteristics  and  the  signatures  of  the  target  and  surrounding  distur- 


bances. 
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2.  Assodate24  a  hk  probability  as  a  function  of  the  miss  distance  found 
and  multiply  the  result  by  the  allegiance  factor  (POE  or  ITO). 

3.  Cumulate  the  results  as  in  the  standard  POE  and  ITO  measure  compu¬ 
tation;  that  is,  for  a  given  class  of  scenario  and  database  conditions, 
e.g.,  a  given  time  period,  geographical  area  and  WAPS  node. 

The  first-order  approximation  is  a  counting  process  that  does  not  account 
for  multiple  CUAs  intersecting  the  WUA.  It  is  expected  that  the  processes  would 
deliver  similar  results  over  sample  sizes  in  excess  of  100,  with  as  little  as  1/1000  as 
many  computing  operations,  a  smaller  memory  requirement  and  shorter  code.  A 
second-order  approximation  can  easily  be  designed  for  in-the-field  quasi-real-time 
post-exerdse  analyses.  A  possible  modification  to  the  first  order  may  include  re- 
pladng  in  Step  2  the  probability  computation  for  a  single  CUA  by  one  for  multiple 
CUAs  intersecting  the  WUA  based  on  the  multiple  miss  distances  found  and  by  mul¬ 
tiplying  the  result  by  the  allegiance  factor  of  the  winning  target  (this  requires  the 
generation  of  a  random  outcome  based  on  the  probabilities  found). 

5.6  Looking-glasses 

Measures  can  be  compared  to  looking-glasses:  they  provide  a  view  of  the 
universe  that  is  limited  in  the  same  way  as  is  the  image  that  a  specific  mirror  can 
produce.  If  one  delivers  an  image  that  has  a  large  field  of  view,  objects  look 
smaller  and  farther  away  than  they  really  are  (as  in  a  car’s  right  rear-view  mirror 
in  North  America).  Small  objects  are  not  noticeable  at  all,  although  the  observer 
may  know  that  they  are  there.  The  opposite  situation  is  the  looking-glass  that 
magnifies  objects  but  presents  a  smaller  field  of  view.  It  is  convenient  when  we 
want  to  focus  on  spedfic  objects  that  are  either  small  or  far  away.  Even  the  per¬ 
fectly  flat  mirror,  which  does  not  distort  the  image,  still  has  a  limited  field  of 
view. 


It  can  be  inversely  proportional  to  the  miss-distance  with  a  threshold  at  the  sum  of  the  CUA  radius  and  the 
WUA  radial  value  at  this  angle  or  its  mean  radius.  It  is  zero  at  the  threshold  and  beyond  it. 


UNCLASSIFIED 

50 


In  the  measures  presented  above  we  can  consider  the  intersection  rates  as  a 
flat  mirror,  the  POE  as  a  magnifying  glass  and  the  ITO  as  one  vnth  a  certain  ability 
to  magnify  the  usefulness  of  information  to  decision  making,  and  thus  to  increase 
mission  effectiveness.  This  is  demonstrated  by  the  follov/ing  estimates  based  on 
these  measures. 

Assuming  that  information  management  (IM)  has  similar  effects  at  each 
node,  we  can  deduce  several  things  about  system  performance  from  Table  VII.  Be¬ 
cause  of  the  architecture  used  during  the  military  exerdse  studied,  at  the  TWCS 
and  FOTC  nodes  the  normalized  decreases  ((FOTC  -  FOTC  with  delay)  /  FOTC))  in 
intersection  rates  are  an  indication  of  the  time  degradation  resulting  from  infor¬ 
mation  management,  development  of  the  hostile  surface  picture  and  all  the  proc¬ 
essing  from  sensors  to  the  WAPS  database,  at  report  time.  We  vrill  apply  the  same 
rationale  to  the  POE  and  ITO  results. 

Assuming  that  the  track  reports  for  contacts  beyond  SAG1  and  SAG2  sensor 
ranges  found  in  these  databases  rely  on  communications  from  the  FOTC,  the  effect 
of  communications  on  mission  effectiveness  can  be  estimated  by  the  normalized 
difference  ((FOTC  delay  -  SAG  delay)  /  SAG  no  delay)  of  the  results  between  the 
FOTC  and  a  SAG.  The  estimate  based  on  the  intersection  rates  does  not  correspond 
exactly  to  the  same  set  of  data  since  it  includes  all  intersections,  whereas  the  POE 
and  ITO  results  are  limited  to  the  sets  for  OTH-T  for  ranges  from  a  SAG  from 
100  km  to  900  km. 
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TABLE  XII 

Cost  in  information  management  and  information  exchange 
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Previously,  we  determined  that  the  sensor  data  were  fairly  accurate,  ac¬ 
counting  for  9%  of  intersection  misses.  In  Table  XII  we  find  evidence  that  IM  at  the 
TWCS  and  FOTC  nodes  is  more  costly  in  terms  of  OTH-T  mission  effectivenesses 
than  in  terms  of  the  positional  accuracy  of  the  reports  (intersection  rates):  about 
61.2%  versus  31.5%,  respectively.  In  estimating  the  effects  of  communications  on 
the  rates,  the  POE  and  the  ITO  at  SAG1  indicate  no  distinctions  between  these 
measures  but  a  small  distortion  for  SAG2.  A  possible  explanation  for  this  is  the  fact 
that  less  data  were  transferred  to  SAG2  compared  to  SAG1,  due  to  the  HIT  broad¬ 
cast  originating  at  the  FOTC.  Consequently,  a  smaller  segment  of  the  data  at  SAG2 
depends  on  the  data  received  from  the  FOTC;  SAG2  relies  more  on  Link  or  local 
data  than  SAG1. 

For  the  experimental  data  studied  in  terms  of  the  hostile  ship  picture  we 
obtain  conclusions  that  are  consistent  with  those  reported  in  Ref.  12  for  an  earlier 
version  of  the  MBMs.  They  are  summarized  in  the  next  chapter. 


With  the  SEM  of  the  MBMs  we  reported  55%  instead  of  62%.  This  is  due  to  the  differences  explained  earlier 
about  the  larger  penalties  for  hitting  the  wrong  ship.  Consequently,  the  new  MBMs  have  more  magnifying 
power  in  terms  of  the  OTH-T  mission  effectiveness  than  the  previous  ones. 
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5.6. 1  Impact  of  Data  Aging 

Using  the  same  strategy  for  assessing  the  impact  of  data  aging  on  the  ITO 
measure  or  OTH-T  mission  success  rate,  we  find  a  relative  loss  of  66.5%  over 
25,7  minutes  if  we  combine  the  results  for  all  the  nodes  (TWCS,  FOTC  and  two 
SAGS).  For  the  SEM  it  is  60%.  These  are  equivalent  to  a  loss  of  information  value  of 
12.9%  per  five  minutes,  if  we  assume  that  it  decreases  linearly  from  the  initial 
time  each  report  is  generated.  For  the  SEM  it  is  11.6%  per  five  minutes  for  all  the 
nodes.  At  the  FOTC  we  observe  a  shorter  mean  delay  of  19.5  minutes  and  conclude 
for  ITO  62.9%  and  SEM  55%  degradations  that  the  impacts  of  information  aging  are 
16%  and  14%  per  5  minutes,  respectively.  The  difference  between  these  results  and 
the  previously  reported  values  is  primarily  due  to  the  assumption  of  a  25-minute 
mean  delay,  rather  than  the  20-minute  delay  reported  here. 

We  do  not  expect  the  rate  of  the  impact  of  data  aging  to  be  linear  (Aimpact 
/  Aage  is  not  linear).  It  is  likely  to  follow  a  Rayleigh  density  function,  vrith  the 
value  of  the  delay  resulting  in  the  maximum  data-aging  impact  rate  depending  on 
the  parameters  of  a  MBM.  We  intend  to  investigate  this  aspect  vrith  the  same  data 
and  models  but  by  sorting  the  results  as  a  function  of  the  report  delay  for  two  sets 
of  MBM  parameters.  Goodness-of-fit  tests  can  be  used  to  assess  the  soundness  of 
our  hypotheses.  We  expect  the  maximum  rate  of  data-aging-impact  to  occur  at  a 
delay  close  to  9  minutes  for  the  current  parameters.  One  may  consider  this  value 
to  be  the  critical  information  age  for  a  dedsion,  task  or  mission  for  a  given  success 
rate. 


5.7  Causality 

Although  links  between  MOPs  and  MOEs  are  implied  by  the  ITO  and  POE  MBM 
definitions  given  in  Chapter  2,  little  has  been  said  about  the  spedfic  MOPs  that  are 
sensed  by  the  six  MBM  models  used.  A  tentative  identification  follows  of  the  sensi¬ 
tivity  of  MBM  models  to  certain  MOPs:  their  ability  to  link  a  particular  MOP  to  the 
OTH-T  effectiveness  of  a  warship. 
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5.7.1  Relevant  MOPs 

International  organizations  are  attempting  to  define  standard  sets  of  MOPs 
to  assess  command  and  control  information  systems  and  to  allow  fair  comparisons 
of  test  results,  but  there  is  still  no  recognized  standard  definition  and  "code-of- 
practice.”  The  MOPs  summarized  in  Table  XIII  are,  in  the  opinion  of  one  of  the 
authors,  those  needed  to  demonstrate  the  value  of  MBMs. 

5.7.2  Examples  of  Causality  Dependence  of  MBAAs 

In  Table  XIV  we  show  that  a  combination  (relative  or  differential)  of  Models 
1  and  2  at  the  FOTC  is  a  good  causal  indicator  of  the  impact  of  information  cur¬ 
rency,  management  delay  and  timeliness  for  OTH-T.  The  combination  of  Models  1 
and  4  is  an  indicator  of  identification  correctness.  Model  4  at  the  FOTC,  relative  to 
Model  4  at  a  SAG,  is  a  good  causal  indicator  of  information  coherency,  but  with 
particular  emphasis  on  the  information  exchange  loss  between  the  FOTC  and  the 
SAG,  while  the  Model  5  combinations  for  these  databases  give  indications  of  infor¬ 
mation  coherency,  currency  and  timeliness  but  with  emphasis  on  information- 
exchange  delay. 

Such  mapping  of  MBM  dependence  and  indications  of  the  impact  of  MOPs  on 
the  mission  success  rates  needs  further  investigation  and  will  be  the  subject  of  fur¬ 
ther  research. 
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TABLE  XIII 

Selected  measures  of  performance 


MOP _ description _ 

allegiance  cor-  The  probability  that  the  allegiance  assigned  to  the  contact  is  correct.  This  is  a 
rectness  condition  for  identification  correctness. 

Identification  The  probability  that  the  unique  Identification  of  the  contact  is  correct, 
correctness 

information  Information  or  picture  coherency  among  participating  units  indicates  the  per- 
coherency  centage  of  data  shared  simultaneously  within  a  given  time  window  and  geo¬ 

graphical  area,  for  a  particular  type  of  data  required  for  a  task. 

information  Information  or  picture  completeness  at  each  database  is  the  percentage  of 
completeness  data  that  is  correct  and  timely,  relative  to  the  available  data  from  all  sources 
(ideally  the  ground  truth).  It  may  be  defined  for  a  particular  task. 

information  Information  or  picture  conciseness  is  the  quantity  of  data  in  a  database  that  is 

conciseness  "free  from  all  elaboration  and  superfluous  detail”  divided  by  the  total  number 

of  data  items.  The  maximum  value  is  one. 

information  Information  currency  is  inversely  proportional  to  data  aging.  The  age  of  the 
currency  data  is  the  elapsed  time  since  last  repxjrt  before  consideration  (sampling  time). 

It  may  include  time  delays  from  any  of  the  processing  stages.  Once  related  to  a 
required  decision  or  task  time  for  a  given  success  rate  it  becomes  the  timeli¬ 
ness  of  that  information  for  an  opportunity. 

information  The  time  required  for  exchanging  information  from  one  database  to  another.  It 
exchange  delay  impacts  the  inter-database  coherency  and  currency. 

information  The  amount  of  information  loss  from  one  database  to  another.  It  impacts  the 
exchange  loss  inter-database  coherency. 

information  The  time  required  to  process  a  contact  report  from  the  time  it  is  generated  by 

management  the  source  or  sensor  to  the  time  it  enters  the  local  database  and  becomes  ac- 

delay  cessible  for  decision  making. 

position  accu-  The  relative  or  geographical  precision  of  the  position  of  a  contact  (may  include 
racy  estimations  of  speed,  altitude  and  attitude).  It  is  usually  expressed  in  terms  of 

areas  or  values  of  uncertainty. 

timeliness  Timeliness  is  how  timely  a  piece  of  information  is  for  a  task  or  decision.  To  be 

distinguished  from  information  currency. 
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TABLE  XIV 

Mapping  of  MOPs  on  the  MBA^  assuming  three  levels  of  dependence:  "d" 
for  strong  dependence,  "i"  for  strong  independence  and '  V  for  weak  de¬ 
pendence  ("a"  for  the  FOTC  and  "b”  for  a  SAG). 


causality 

MBM  model  number  or  combination  of  numbers 

POE 

ITO 

combination 

MOP 

1 

2a 

2b 

■ 

5a 

5b 

1a:2a 

1:4 

4a:4b 

5a:5b 

allegiance  cor¬ 
rectness 

■ 

■ 

■ 

■ 

■ 

■ 

i 

i 

■ 

■ 

identification 

correctness 

■ 

i 

■ 

■ 

■ 

■ 

i 

■ 

i 

i 

i 

■ 

i 

i 

i 

i 

i 

d 

■ 

information 

completeness 

■ 

[H 

■ 

i 

■ 

i 

■ 

i 

w 

w 

information 

conciseness 

■ 

■ 

■ 

■ 

i 

M 

i 

information 

currency 

d 

d 

d 

d 

d 

d 

d 

i 

i 

d 

■ 

1 

■ 

1 

■ 

■ 

■ 

mm 

i 

■ 

information 
exchange  loss 

■ 

i 

■ 

i 

■ 

1 

■ 

i 

■ 

w 

information 

management 

delay 

i 

■ 

■ 

■ 

d 

1 

■ 

■ 

■ 

■ 

■ 

■ 

d 

■ 

w 

i 

i 

w 

timeliness 

■ 

■ 

■ 

nil 

m 

d 

m 

i 

i 

d 

Table  XIV  is  a  tentative  mapping  of  MOPs  on  MBMs  and  some  of  their  combi¬ 
nations. 
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6  CONCLUSIONS  AND  RECOAAMENDATIONS 


An  analysis  of  the  results  obtained  with  the  new  DREV  MBMs  for  a  simulated 
military  exerdse  follows,  including  a  comparison  with  the  results  obtained  using 
the  earlier  version  of  the  MBMs.  Only  the  significant  results  that  offer  insight  into 
synthesis  or  systems  issues  are  presented. 

Despite  the  complexity  of  the  systems  evaluated,  the  new  measures  were 
able  to  recognize  changes  and  variations  in  the  systems  and  in  the  opposing  forces 
that  the  current  or  classic  MONIME  measures  did  not  detect  or  identify. 

The  reported  method  and  its  improved  version  thus  offer  better  dis¬ 
crimination  among  system  options  and  reduce  development  risk  by  accu¬ 
rately  estimating  the  impact  that  changes  in  procedures,  systems  and 
information  have  on  mission  effectiveness. 

Results  from  the  new  version  of  DREV  MBMs  (previous  version  results  are  be¬ 
tween  parentheses)  lead  to  the  following  conclusions; 

1 .  Improvements  in  WANTAP  capabilities  versus  hostile  ships  that  can  be 
obtained  by  optimizing  sensor  suites  appear  to  be  limited  to  about  9% 
(was  8%),  and  that  gain  is  possible  only  if  the  WANTAP  systems  them¬ 
selves  are  optimized  first; 

2.  changes  in  WANTAP  systems  and  procedures  can  improve  performance 
in  excess  of  63%  for  the  ITO  (was  55%  for  the  SEM),  from  the  reference 
value  at  sensor  time  at  the  FOTC  node  (assuming  instantaneous  track 
management  and  information  exchange); 

3.  improvements  exceed  77%  (was  70%)  at  the  Surface  Action  Group  1 
(SAG1)  nodes  under  the  same  conditions,  providing  that  communica¬ 
tions  improvements  from  the  FOTC  to  the  SAGs  are  made  (i.e.,  in¬ 
cluding  the  assumptions  from  2);  and  finally, 

4.  communications  impact  the  distributed  picture  by  only  14%  all  cases 
at  SAG1  (was  15%  for  SAG1  intersection  rate),  about  1.5  times  as  much 
as  sensors. 
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The  most  important  step  to  improve  overall  performance  is  better  exploita¬ 
tion  of  existing  sensor  and  source  data  by  improving  WANTAP  systems,  concepts 
and  procedures  at  the  tactical  information  management  node  or  in  a  distributed 
version  of  it. 

Based  on  the  results  obtained  for  hostile  surface  contacts  and  for  the  sur¬ 
veillance  deployment  used  during  the  exercise,  an  improvement  of  4%  in  sensor 
performance  alone,  without  improving  the  WANTAP  systems  and  communications, 
would  provide  less  than  1.3%  (was  1.6%)  improvement  to  the  shared  picture.  This 
apparent  loss  in  exploiting  available  information  may  be  of  concern  to  users  and  to 
experts  such  as  the  members  of  the  Copernicus  Requirements  Working  Group 
(CRWG),  who  are  developing  improved  JMCIS  or  GCCS  and  related  architectures 
and  procedures. 

Another  important  result  from  the  data  examined  using  the  proposed  deci¬ 
sion-based  measures  of  goodness  is  a  simple  rule  of  thumb  that  the  hostile-surface 
WANTAP  usefulness  degrades  by  12.9%  (was  11.6%)  per  5  minutes  of  data  aging  for 
the  four  databases  observed,  vrith  an  average  delay  of  25.7  minutes.  At  the  FOTC 
the  observed  shorter  mean  delay  of  19.5  minutes  with  62.9%  ITO  and  55%  SEM  de¬ 
gradations  implies  that  the  impacts  of  information  aging  here  are  16%  and  14%  per 
5  minutes,  respectively. 

Better  transfer  of  sensor,  intelligence  and  Link  11  data  is  required  to  im¬ 
prove  the  assistance  that  WAPS  provides  for  commander  dedsions.  We  recommend 
that  operators  and  commanders  afloat  be  provided  with  more  effident  data  man¬ 
agement  aids,  including  some  automation  in  the  selection,  exploitation,  fusion  and 
integration  of  Link  11  data. 

Investigations  of  the  effectiveness  of  dead  reckoning  show  that  using  this  es¬ 
timator  can  decrease  the  value  of  the  displayed  hostile  picture  by  as  much  as  67%. 
Consequently,  we  strongly  recommend  that  its  use  be  avoided,  except  to  display 
merchant-ship  information. 

Using  these  MBMs  significantly  reduced  the  dependency  of  measured  val- 
ues-the  results  of  the  study— on  the  actual  detailed  decisions-how  surveillance 
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assets  were  deployed,  for  example-made  during  the  exerdse.  By  simulating  ded- 
sions  applied  to  each  valid  scenario,  the  proposed  measures  of  goodness  were  also 
shown  to  be  independent  of  the  style  or  training  of  decision  makers. 

This  report  demonstrates  the  feasibility  of  statistically  estimating  the  im¬ 
pact  of  information  and  systems  quality  on  dedsion  outcomes  and  mission  effec¬ 
tiveness.  Generally  speaking,  raw  data  that  directly  relates  the  quality  of  com¬ 
mand  information  systems  to  measures  of  effectiveness  are  sparse  and  expensive 
to  obtain,  so  it  is  very  difficult  to  demonstrate  information  and  systems  qualities 
and  their  impacts  on  dedsion  outcomes  and  mission  effectiveness.  DREV  model- 
based  measures  address  this  difficult  problem.  They  improve  estimation  accuracy 
using  the  same  raw  data  currently  available  from  instrumented  military  exerdses. 
Both  versions  of  MBMs  provided  insights  that  were  not  available  with  the  measures 
currently  applied  to  such  data.  This  can  be  confirmed  by  comparing  our  results 
with  those  in  Refs.  9-11,  14-21. 

For  the  hostile-ship  picture  found  in  a  commander's  database,  we  have  de¬ 
fined  two  model-based  measures  to  capture  the  essential  components  of  the  im¬ 
pact  of  the  information  used  in  engagement  dedsions:  the  POE  for  the  pertinence 
of  engagements  and  the  ITO  for  the  intended  target  opportunity.  The  ITO  measure 
amplifies  the  POE’s  significance  by  assigning  a  smaller  reward  if  the  hostile  target 
that  is  intercepted  is  not  the  intended  one.  Both  measures  use  a  fixed  cost  for 
launching  a  missile  and  both  compute  interception  probabilities  and  reward  values. 
In  both  cases,  hypothetical  blue-on-blue  engagements  are  assigned  large  negative 
values  to  indicate  the  counter-productivity  of  such  actions.  Over  a  large  number  of 
engagements  these  measures  provide  a  fair  evaluation  of  the  goodness  of  the  pic¬ 
ture  used  in  the  dedsion  process. 

Our  approach  to  measure  the  impact  of  MOPs  (measures  of  performance  of 
functions  or  systems)  causally  on  MOEs  (measures  of  effectiveness  for  mission 
goals)  suggests  the  follovring  analogy:  if  the  results  (MOEs)  of  dedsions  and  mis¬ 
sions  are  identified  as  the  effects,  and  the  picture  and  system  qualities  (MOPs)  as 
causes,  then  one  can  say  that  we  causally  measure  the  value,  usefulness  or  good¬ 
ness  of  a  tactical  picture  and  a  C3IS  to  commanders’  dedsions  and  missions. 
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The  specialization  to  the  over-the-horizon  hostile  picture  increases  the  dis¬ 
crimination  capadty  of  the  measures  proposed  by  focusing  on  the  value  of  this  part 
of  the  picture  for  commanders’  OTH-T.  However,  DREV  MBMs  are  not  limited  to 
this  aspect:  they  can  be  adapted  for  the  estimation  of  types  of  contacts  or  condi¬ 
tions  identifiable  in  the  raw  data  from  instrumented  military  exerdses  for  corre¬ 
sponding  types  of  actions  or  missions.  We  just  need  to  adapt  the  model-based 
measures  to  what  we  want  to  measure,  predsely  and  accurately. 

For  example,  the  proposed  models  accurately  measure  the  relative  im¬ 
provements  due  to  changes,  but  since  we  do  not  have  a  well-established  level  of 
picture  quality  needed  for  a  particular  mission,  the  measures  cannot  deliver  an  ab¬ 
solute  measure  of  goodness.  In  addition,  they  do  not  assess  the  damage  inflicted  to 
hostile  ships.  On  the  other  hand,  from  the  application  of  these  measures  to  a  set  of 
several  exerdses  (at  least  four),  it  would  be  possible  to  infer  the  minimum  infor¬ 
mation  quality  and  systems  performance  required  for  that  spedfic  type  of  dedsion 
or  mission.  This  minimum  requirement  could  be  expressed  in  terms  of  ac¬ 
tion/mission  success  rates  or  risks  of  failure,  giving  a  probabilistic  indication  that 
at  this  information  quality  level,  for  a  given  class  of  mission  scenarios,  mission  suc¬ 
cess  is  in  excess  of  a  given  percentage,  say  80%. 

A  MBM  for  the  WANTAP  completeness  measure  has  been  proposed  and  a 
clear  distinction  shown  between  the  proposed  measures  and  traditional  ones:  the 
new  measures  are  geared  at  evaluating  the  effectiveness  of  a  WANTAP  for  a  spe¬ 
dfic  mission,  rather  than  at  investigating  some  particular  system  performance  pa¬ 
rameter  such  as  communication  delay.  Further  studies  are  needed  to  explore 
model-based  measures  for  other  mission  goals  such  as  drug  interdiction  or  area 
control. 
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ambiguity 

ambiguous  track 

assumed  friend 

attribute 

causal  relation 
classification 

constructed  track 

containment  ellipse 


APPENDIX  A 
GLOSSARY 

A  sensor  report  that  a  wide-area  correlator/tracker  (C/T) 
system  has  processed  but  has  been  unable  to  place  into  a 
constructed  track.  The  inability  to  determine  if  sensor  re¬ 
ports  are  for  a  new  track  or  are  updates  to  an  existing  one. 

A  track  that  cannot  be  automatically  assodated  and  fused 
with  a  known  track  in  the  C2  system  track  database.  It  may 
be  a  new  track  or  it  may  be  a  known  one  that  cannot  be  as¬ 
sodated  because  the  POSIT  report  contains  corrupted,  in¬ 
correct  or  insufficient  information.  Ambiguous  tracks  must 
be  dealt  with  manually  by  the  C2  system  operator. 

A  track  that  is  assumed  to  be  a  friend  because  of  its  charac¬ 
teristics,  behavior  or  origin. 

Any  piece  of  information  that  increases  the  knowledge  of  a 
contact  under  surveillance. 

Relationship  that  directly  links  effects  to  their  causes. 

In  surveillance,  the  process  of  assigning  an  agreed-upon  at¬ 
tribute  such  as  allegiance  for  operations,  e.g.,  neutral,  hos¬ 
tile  or  faker. 

The  subset  of  sensor  reports  considered  by  a  C/T  to  belong 
to  the  same  object,  platform  or  track. 

A  probability  ellipse  indicating  the  area  where  a  contact  (for 
example  a  ship)  is  located,  with  a  given  probability,  e.g., 
0.8  or  80%. 
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correlated-track 

data 

correlation 

data  fusion 


dead  reckoning 

duplicated  track 

engage 

extraneous  track 


Data  resulting  from  correlation  or  tracking,  or  both. 


Level  of  closeness  among  a  subset  of  sensor  reports  that  Is 
used  to  assign  reports  to  a  given  contact.  The  level  of  confi¬ 
dence  of  contact  Information  Increases  as  further  statisti¬ 
cally  Independent  reports  are  correlated. 

The  combining  of  data  from  multiple  sources,  or  from  the 
same  source  over  a  period  of  time,  into  a  unified  represen¬ 
tation  of  the  entitles  (units  and  events)  described  by  those 
sources.  The  fused  Information  represents  an  entity  In 
greater  detail  and  with  less  uncertainty  than  Is  obtainable 
from  any  of  the  original  sources.  Data  fusion  includes  the 
processes  of  alignment,  correlation,  association  and  combi¬ 
nation. 

Positional  estimation  of  a  mobile  unit  at  a  future  time  from 
the  position  and  speed  observed  at  sensor  time,  assuming  no 
maneuvering;  a  tactical  and  navigation  aid. 

The  situation  In  which  a  single  observed  unit  gives  rise  to 
two  separate  tracks  In  the  C2  system  database.  Duplicated 
tracks  must  generally  be  resolved  manually  by  the  C2  sys¬ 
tem  operator.  Misspelling  of  unit  names  Is  a  typical  cause. 

To  take  action,  such  as  firing  a  missile,  against  a  target  that 
has  been  Identified  "hostile”  In  the  tactical  database. 

A  track  in  the  C2  system  track  database  for  which  there  Is 
no  corresponding  real  unit.  It  Is  often  made  up  of  segments 
of  tracks  from  several  real  units,  espedally  if  these  units  are 
moving  In  convoy. 


faker 


A  friendly  track  acting  as  a  hostile  for  exerdse  purposes. 
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friend 

farthest-on-circle 

ground-truth  track 

hostile 

identification 

interoperability 


joker 

"kilo” 

level  of  significance 


A  track  belonging  to  a  declared  friendly  unit  or  nation. 

A  dedsion-aid  drcle  that  expands  from  a  reported  contact 
position  and  time  at  a  selected  speed;  it  is  used  to  estimate 
the  area  within  which  the  contact  might  be  at  subsequent 
times. 

Track  information  that  is  used  as  a  reference  for  other  sys¬ 
tems;  it  may  include  an  accurate  position  of  the  platform. 

A  track  that  is  eligible  to  be  engaged. 

In  surveillance,  determination  of  an  identity  attribute  by 
source  data,  intelligence  reports  or  from  the  behavior  of  the 
object(s)  of  a  track. 

Capabilities  of  systems,  functions,  people  and  information 
that  allow  effective  interaction  and  joint  operations.  Sev¬ 
eral  international  organizations  spend  considerable  effort  to 
improve  and  maintain  this  capability  across  different  meta¬ 
systems.  The  focus  in  the  context  of  this  report  is  on  infor¬ 
mation,  functions,  systems,  procedures,  protocols  and  their 
management  in  a  distributed  situation  for  WANTAP  compila¬ 
tion. 

A  friendly  track  acting  as  a  "suspect”  for  exerdse  purposes. 
A  friendly  track  of  spedal  interest. 

For  one  test  or  a  set  of  statistical  tests,  the  rate  or  prob¬ 
ability  at  which  statistical  test  results  reject  the  hypothe¬ 
sized  probabilistic  model  (by  chance  only)  when  the  model 
is  actually  true. 
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link 


Link  [xx] 


local-area  picture 

(LAP) 


maritime-tactical 
picture  (MTP) 


Any  means  of  communications  between  system  components 
or  users,  mainly  radio  and  satellite  communications  for 
WANTAP-related  information  systems. 

One  of  the  NATO  military  radio  communications  standards 
for  data  and  related  functions  (data  dictionaries,  message 
formats,  procedures,  communications  protocols,  coding, 
modulation).  Examples  include  Link  11  (equivalent  to  the 
USN’s  TADIL-A  or  NTDS),  Link  16  and  Link  4. 

The  picture  developed  by  the  ship's  own  sensors,  i.e.,  or¬ 
ganic  information.  This  information  is  controlled  and  col¬ 
lected  by  assets  under  the  ship's  command  direct  control 
and  is  usually  real-time.  Under  Link  11,  combining  the  in¬ 
formation  from  all  participants  in  a  given  Link  11  network 
extends  the  LAP  area.  This  effectively  increases  the  horizon 
of  each  ship's  organic  picture. 

The  picture  (tactical  information)  necessary  to  support  Al¬ 
lied  maritime  operations  over  the  area  of  interest  (AOI)  of 
the  maritime  force  commander  (CTF).  This  may  range  from 
a  spedfic  ocean  area  to  an  overall  global  picture.  The  MTP 
is  a  combination  of  the  local-area  picture  (LAP)  and  the 
wide-area  picture  (WAP).  The  MTP  is  the  common  (or  recog¬ 
nized)  picture  that  includes  both  organic  and  non-organic 
information.  Future  systems  will  integrate  all  this  informa¬ 
tion  and  the  distinction  between  the  LAP  and  the  WAP  will 
disappear.  The  need  to  distinguish  between  organic  versus 
non-organic  information  vrill  be  reduced. 
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metasystems 


neutral 


non-organic  asset, 
source  or  sensor 


non-organic  informa¬ 
tion 

non-unique  ID  at¬ 
tribute 

nonparametric 


Systems  made  up  of  several  systems  or  the  logical  and  func¬ 
tional  equivalent  of  a  complex  combination  of  system-to- 
human  interactions.  It  also  refers  to  the  sdentific  applica¬ 
tion  of  systems  methodologies  that  address  systems  in 
breadth  and  depth  for  all  strata  and  aspects.  From  a  human 
factor  aspect,  it  studies  multi-phase,  multi-level  and  multi- 
paradigmatic  creative  problem-solving  process  for  individu¬ 
als,  often  in  large  groups.  One  primary  goal  is  to  study,  de¬ 
sign  and  then  implement,  in  a  cost-effective  fashion,  me¬ 
tasystems  that  are  user-responsive. 

A  track  whose  characteristics,  behavior,  origin  or  nationality 
indicate  that  it  is  neither  supporting  nor  opposing  friendly 
forces. 

Assets  that  are  not  under  the  direct  control  of  a  given  com¬ 
mand  organization,  as  opposed  to  organic  assets,  which  are 
under  its  direct  control. 

Information  collected  by  non-organic  assets  (sensors  and 
agents).  It  comes  from  sources  not  under  the  direct  control 
of  the  commander  at  the  command  level  under  considera¬ 
tion. 

An  ID  that  is  unique  at  a  high  level  for  which  low-level  at¬ 
tributes  correspond  to  several  units. 

A  statistical  test  that  does  not  require  the  estimation  of  the 
parameters  or  distribution  of  the  hypothesized  population. 
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order  statistic 


organic  asset,  source 
or  sensor 

organic  information 

picture  quality  (PQ) 

procedure 

rule 


A  statistic  that  requires  the  observations  to  be  ordered  (by 
size,  color,  attribute,  alphabetically,  numerically,  etc.). 
The  ordered  sequence  is  then  compared  with  the  hypothe¬ 
sized  sequence  order.  This  comparison,  using  a  given  algo¬ 
rithm,  forms  one  of  the  many  order  statistics.  For  simple 
hypotheses  and  small  sample  sizes,  exact  significance  levels 
can  be  obtained  with  tests  based  on  binomial  or  multinomial 
laws. 

Assets  under  direct  operational  control  of  a  given  command 
organization. 

Information  from  sources  under  the  control  of  a  commander 
at  a  given  level. 

A  measure  of  (i)  the  completeness  of  the  picture,  (ii)  the 
positional  accuracy  of  the  tracks,  (iii)  the  correctness  of  the 
track  attributes,  (iv)  the  timeliness  of  the  track  reports,  and 
(v)  the  absence  of  extraneous,  ambiguous  and  duplicated 
tracks. 

The  manner  in  which  particular  courses  of  action  are  taken 
to  achieve  an  objective.  In  general  they  contain  all  of  the 
steps  that  must  be  followed,  whether  manually  or 
automatically,  to  ensure  that  the  correct  action  is  taken,  in 
accord  with  the  rules  on  which  they  are  based. 

The  prindples  or  regulations  on  which  procedures  are 
formulated.  They  must  account  for  the  conditions  that  are 
likely  to  be  faced  when  practicing  procedures.  The  rules 
that  are  formulated  for  future  command,  control, 
communications  and  intelligence  (C3I)  systems  will  have  to 
account  for  all  the  conditions  that  determine  parameters 
such  as  track  quality,  confidence  levels,  classification,  track 
numbering  and  a  host  of  other  factors  pertinent  to  the 
target  acquisition  and  fusion  processes. 
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sensor  report 


statistical  inference 


suspect 


target 


track  and  tracking 


track  handover 


unique  ID  attribute 


A  data  report  generated  by  a  sensor  or  an  information 
source  pertaining  to  a  contact,  platform  or  group  of  plat¬ 
forms. 

The  process  of  extrapolating  from  statistics  on  a  sample  to 
the  whole  population  of  possible  samples,  at  a  given  signifi¬ 
cance  level. 

A  track  that  is  potentially  hostile  (because  of  its  character¬ 
istics,  behavior,  origin  or  nationality). 

A  contact  report  that  is  the  object  of  surveillance  based  on 
an  algorithm,  automatic  or  manual.  A  qualification  given  to 
a  contact  that  is  under  the  scrutiny  of  an  observer,  an  algo¬ 
rithm  or  a  process  necessary  to  develop  a  clear  MTP.  A  con¬ 
tact  target  may  have  a  large  variety  of  possible  identities  in 
a  real  system,  as  compared  to  the  spedfic  identity  (neutral, 
friend,  hostile)  attributed  in  a  ground-truth  file. 

Exploitation  of  a  sequence  of  reports  from  one  or  several 
spatially  distributed  sensors  or  other  sources  of  information, 
to  deduce  a  contact  position  at  an  arbitrary  time  in  the 
past,  present  and  future. 

In  data  fusion,  the  process  of  transferring  the  responsibility 
for  produdng  a  track  from  one  unit  to  another  (human  op¬ 
erators  and/or  processes). 

An  attribute  that  is  suffident  to  identify  the  class  to  the 
unit  level. 


unknown 


An  evaluated  track  which  has  not  yet  been  identified. 
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wide-area  picture 


zombie 


The  picture  developed  using  assets  not  under  CTF  direct 
control,  e.g.,  shore-based  assets  and  space-borne  sensors. 
The  volume  of  this  information  was  once  small,  but  it  has 
increased  over  time  to  the  point  that  spedal  information 
dissemination  architectures  are  now  required,  e.g.,  the  cur¬ 
rent  USN  Force  Over-the-horizon  Track  Coordinator  (FOTC) 
used  with  JMCIS-Afloat  and  the  Officer  in  Tactical  Command 
Information  exchange  System  (OTCIXS). 

A  "suspect”  air  track  conforming  to  ATC  rules  or  flying  a 
recognized  traffic  path. 
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APPENDIX  B 

ACRONYMS  AND  ABBREVIATIONS 


AAW  anti-ai’r  warfare^ 

AAWC  Anti-air  Warfare  Commander 

ACIXS  Allied  Command  Information  Exchange  System++ 

AHWG  Ad-hoc  Working  Group 

AMIME  AUS-CAN-NZ-UK-US  Ad-hoc  Working  Group  on  the  management 

of  organic  and  non-organic  information  in  a  maritime  environ¬ 
ment 

AOI  area  of  operational  interest 

AOU  area  of  uncertainty 

ARE  Admiralty  Research  Establishment,  now  DRA  (UK) 

AREC  Air  Resource  Element  Commander 

ASTAB  automatic  status  board 

ASUW  anti -surface  warfare 

ASUWC  Anti-^rface  Warfare  Commander 

ASW  antisubmarine  warfare 

ASWC  Antisubmarine  Warfare  Commander 

ATC  air  traffic  control 

AUS  Australia 

AUS-CAN-NZ-UK-  Counterpart  of  the  NATO  TTCP  information  exchange  agreement 
US  for  operational  and  in-development  systems,  related  standards 

and  problems.  Member  countries:  Australia,  Canada,  New  Zea¬ 
land,  United  Kingdom  and  United  States, 
b-pdf  bivariate-probability  density  function 

BG  Battle  Group 

BIOS  beyond  line-of-sight  (radio-wave  propagation) 

C/T  correlator/ tracker 

C2  command  and  control 

C2IR  Command  And  Control  Information  Requirement 

(a  UK-NISAS  tool) 

C3  command,  control  and  communications 


Capitals  are  used  only  when  required,  e.g.,  for  the  name  of  a  specific  system  or  group. 
Underlining  is  used  only  when  the  result  does  not  follow  a  simple  abbreviation  rule. 
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C3I 

C3IS 

CAN 

CAS 

CDS 

CIS 

civ 

COF 

COI 

COP 

CRWG 

CS 

CTF 

CTG 

CUA 

CVBG 

CWC 

DLRP 

DMCS 

dmi 

DMSS 

DR 

DRA 

DREV 

edc 

ELINT 

ESM 

FDDS 

FOSP 

FOTC 

FSM 

GCCS 

GDFS 

GEM 

GEO 

GRW 

GT 


command,  control,  communications  and  intelligence 
communications,  command,  control  and  information  system 
Canada 

center  of  the  area  to  be  scrutinized 

combat  direction  system 

communications  and  information  system 

convolution-integration  value 

cost  of  firing 

contact  of  interest 

common  operating  picture 

Copernicus  Requirements  Working  Group  (US) 

commander  of  an  armed  ship,  or  the  ship  effecting  the  weapon 

engagement  in  OTH-T 

Commander  Task  Force 

Commander  Task  Group 

drcular-uncertainty  area 

Carrier  Vehicle  Battle  Group  (US) 

Composite  Warfare  Commander 
data  link  reference  position 

Director  Maritime  Combat  Systems  (CAN),  now  DMSS 
data  mile,  defined  as  6  000  ft  (1  828.8  m) 

Director  Maritime  Ship  Support  (CAN) 
dead  reckoning 

Defence  Research  Agency  (UK) 

Defence  Research  Establishment  Valcartier  (CAN) 
error  detection  code  (sometimes  used  for  error  detection  and 
correction  code), 
dectronic  intelligence 

electronic  support  measures  /  warfare  support  measures 
Flag  Data  Display  System  (US) 

Fleet  Ocean  Surveillance  Product  (UK) 

Force  Over-the-horizon  Track  Coordinator  (US) 
finite-state  machine 

Global  Command  and  Control  System  (US) 

Graphic  Data  Fusion  System  (US) 
general-engagement  measure 
general-engagement  opportunity 
general-reward  value 
ground-truth 


UNCUSSIFIED 

73 


HB5 

HB5D 

HEAT 

HF 

HIT 

HP 

HQ 

ID 


IDU 

IEEE 

lERs 

lES 

IH 

IM 

ISAR 

ISAT 

ISO 

ITO 

ITO-D 

ITO-D-P 

ITO-0 

JMCIS 

JMCIS 

JOTS 

UP 

Link  [xx] 


LIVEX 

LOB 

LOS 

A^ARCOT 

MBM 


Handbook  5  planned  by  AUS-CAN-NZ-UK-US 
Handbook  5  Draft 

Headquarters  Effectiveness  Assessment  Tool  (US) 

high  frequency 

high  interest  tracks 

hit  probability 

headquarters 

identification.  Preferably  unique  for  a  given  system  or  command 
stratum,  since  each  stratum  is  distributed  but  different  codes 
can  exist  among  strata.  It  is  considered  a  track  attribute  al¬ 
though  it  serves  no  track  or  contact-report  function  other  than 
management, 
interface-data  unit 

Institute  of  Electrical  and  Electronics  Engineers 

information  exchange  requirements 

ideal  engagement  situation 

intended  hostile 

information  management 

Inverse  Synthetic  Aperture  Radar  (US) 

Information  Systems  Architecture  Tool  (UK-NISAS) 

International  Organization  for  Standardization 

intended  target  opportunity 

ITO-delay 

ITO-delay-prediction 
ITO  optimal 

Joint  Maritime  Command  Information  Strategy  (CAN) 

Joint  Maritime  Command  Information  System  (US),  it  includes 
NTCS-A  and  some  interfaces. 

Joint  Operational  and  Tactical  System  (US),  JOTS  II  is  the  core 
program  in  NTCS-A. 
local-area  picture 

A  NATO  military  radio  communication  standard  for  data  and  re¬ 
lated  functions,  e.g..  Link  11  (equivalent  to  USN’s  TADIL-A  or 
NTDS),  Link  16,  Link  4 
live  exerdse 
line  of  bearing 

line-of-sight  (radio-v/ave  propagation) 

Maritime  Command  Operational  Training  exercise  (CAN) 
model-based  measure  /  mesure  basee  sur  des  modeles 
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MCOIN 

MDE 

MDP 

mi 

MIDAS 

min 

MIST 

MMI 

MOE 

MONIME 

MOP 

MSC 

MTP 

MTST 

N-KRS 

NATO 

Navcomms 

NCCOSC 

NDHQ 

NISAS 

nmi 

NOSC 

NRAD 

NTCS-A 

NTDS 

NZ 

OPFOR 

OSI 

OTCIXS 

OTG 

OTH-DCaT 

OTH-T 

OTCIXS 

PC 

pdf 

ph 

phn 

phs 


Maritime  Command  information  system  (CAN) 

mesure  d’efficadte 

mesure  de  performance 

statute  mite:  5  280  ft  (1  609.35  m) 

Maritime  Intelligence  Dissemination  and  Analysis  System 
minute 

Maritime  Intelligence  Support  Terminal  (AUS) 
man-machine  interface 
measure  of  effectiveness  for  a  mission  goal 
Management  of  Organic  and  Non-organic  information  in  a  Mari¬ 
time  Environment,  a  program 
measure  of  performance  of  a  function  or  system 
mission  success  criteria 
maritime  tactical  picture 
maneuvering  target  statistical  tracker 
Naval  Knov/ledge-based  Replanning  Systems  (US) 

North  Atlantic  Treaty  Organization 
Naval  Communications  Committee  of  AUS-CAN-NZ-UK-US 
Naval  Command,  Control  and  Ocean  Surveillance  Center  (US) 
National  Defence  Headquarters  (CAN) 

Naval  Information  Systems  Architecture  Study  (UK) 

international  nautical  mile:  1  852  m 

Naval  Ocean  Systems  Center,  now  NCCOSC  (US) 

Naval  Research  and  Development  (US) 

Naval  Tactical  Command  System  -  Afloat  (US) 

Naval  Tactical  Data  System  (US),  i.e..  Link  11 

New  Zealand 

^posing  forces 

Open  Systems  Interconnection 

Officer  in  Tactical  Command  Information  Exchange  System 
Over-the-horizon  Targeting  Gold  (a  message  format) 
over-the-horizon  detection,  classification  6t  targeting 
over-the-horizon  targeting 

Officer  in  Tactical  Command  Information  Exchange  System 

picture  consistency 

probability  density  function 

probability  of  hitting  a  given  target 

probability  of  hitting  nothing 

probability  of  hitting  something 
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pm 

probability  mass 

POE 

pertinence  of  engagement 

POE-D 

POE-delay 

POE-D-P 

POE-delay-prediction 

POE-0 

POE-optimal 

POST 

Prototype  Ocean  Surveillance  Terminal  (US) 

PQ 

picture  quality 

PRI 

pulse  repetition  interval 

PRV 

pertinence  reward  value 

QIER 

quantified  information  exchange  requirement 

QIR 

quantified  information  requirement 

R&D 

research  and  development 

RAN 

Royal  Australian  Navy 

REPEAT 

J^peatable  Performance  Evaluation  and  Analysis  Tool  (US) 

RESA 

Research,  Evaluation  and  Systems  Analysis  War-gaming  Fadlity 
(US) 

RF 

radio  frequency 

RIMPAC 

Rim  of  the  Padfic  (In  this  document  it  refers  to  a  naval  LIVEX  of 
Padfic  Rim  countries) 

RMP 

recognized  maritime  picture 

RN 

Royal  Navy  (UK) 

ROES 

rule(s)  of  engagements 

SAG 

Surface  Action  Group 

SEDSCAF 

standard  ELINT  data  systems  codes  and  format  (US) 

SEM 

specific  engagement  measure 

SEO 

spedfic  engagement  opportunity 

SEWC 

Space  and  Electronic  Warfare  Commander  (Proposed  by  US) 

SPAWAR 

Space  and  Naval  Warfare  Systems  Command  (US) 

SRW 

specific  reward  value 

STANAG  [XX] 

NATO  standardization  ^reement 

STU 

secure  telephone  unit 

STWC 

Strike  Warfare  Commander 

SUBOPAUTH 

Submarine  Operating  Authority  (US) 

SWAN 

Secure  Wide  Area  Network 

TADIL-A 

Tactical  Data  Information  Link  -  Afloat,  i.e..  Link  1 1 

TADIXS 

Tactical  Data  Information  Exchange  System 

TASM 

Tomahawk  Anti-ship  Missile  (US) 

TC 

track  continuity 

tdv 

total  (value)  of  convolution-integration  values 
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TDP 

TIMEX 

TIMSIM 

TLWR 

TP 

TPQ 

TRV 

TTCP 

TUA 

TWCS 

UK 

US 

USN 

WANTAP 

WANTAP  Task 

WAP 

WAPS 

WGS 

WT-C 

WUA 


tactical  data  processor 

Tactical  Information  Management  Exerdse  (US) 

Tactical  Information  Management  Simulation  (AUS-CAN-NZ-UK- 
US) 

top  level  warfare  requirements  (US) 

track  purity 

tactical  picture  quality 

target  reward  value 

the  Technical  Cooperation  Program 

target-uncertainty  area 

Tomahawk  Weapons  Control  System  (US) 

United  Kingdom 
United  States 
US  Navy 

^de-area  naval  ^ctical  picture 

Analysis  of  Wide-area  Naval  Tactical  Picture  Task  (CAN) 

wide-area  picture 

wide-area  picture  systems  like  JOTS,  TWCS,  JMCIS  and  GCCS 
World  Geodetic  System 
wde-area  tracker-correlator 
weapon-uncertainty  area 
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APPENDIX  C 

SUAAA^RY  TABLES  OF  RESULTS 


Intersection  Rates 
Pertinence-of-engagement  Measure 
Intended-target-opportunity  Measure 
Delays 

Positional  Errors  (all  cases) 
Positional  Errors  (outliers  removed) 
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Intersection  Rates  -  Summary 


WANTAP 

TWCS 

FOTC 

SAG1 

SAG2 

day 

(%) 

(%) 

(%) 

(%) 

1 

98.6 

91.9 

82.9 

85.3 

2 

90.1 

91.2 

88.9 

82.0 

3 

90.5 

68.8 

74.0 

69.1 

4 

89.4 

86.8 

89.1 

87.3 

Overall 

91.1 

84.6 

84.2 

81.1 

TABLE  1  -  TIMSIM  '93  intersection  rate  %  (no  delay) 


WANTAP 

TWCS 

FOTC 

SAG1 

SAG2 

day 

(%) 

(%) 

(%) 

(%) 

1 

42.9 

56.4 

47.2 

47.7 

2 

72.0 

71.3 

53.4 

33.9 

3 

62.7 

41.5 

41.3 

46.6 

4 

61.8 

58.4 

41.6 

51 .6 

Overall 

62.3 

58.0 

45.7 

46.3 

TABLE  2  -  TIMSIM  '93  intersection  rate  %  (delay  without  DR  prediction) 


WANTAP 

TWCS 

FOTC 

SAG1 

SAG2 

day 

(%) 

(%) 

(%) 

(%) 

1 

31.4 

41.8 

46.0 

38.4 

2 

57.4 

55.2 

36.0 

18.2 

3 

47.5 

40.0 

34.3 

34.4 

4 

45.7 

40.4 

26.3 

46.3 

Overall 

47.5 

45.1 

33.8 

36.5 

TABLE  3  -  TIMSIM  '93  intersection  rate  %  (delay  with  DR  prediction) 
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Pertinence-of-engagement  Measure  -  Summary 


WANTAP 

day 

TWCS 
(-2.5,  0.5) 

FOTC 
(-2.5,  0.5) 

SAG1 
(-2.5,  0.5) 

SAG2 
(-2.5,  0.5) 

1 

0.34 

0.37 

0.43 

0.30 

2 

0.32 

0.17 

0.16 

0.12 

3 

0.12 

-0.03 

0.00 

-0.07 

4 

0.10 

0.19 

0.22 

0.15 

Overall 

0.20 

0.16 

0.17 

0.12 

TABLE  1  -TIMSIM  '93  pertinence-of-engagement  measure  (no  delay) 


WANTAP 

day 

TWCS 
(-2.5,  0.5) 

FOTC 
(-2.5,  0.5) 

SAG) 

(-2.5,  0.5) 

SAG2 
(-2.5,  0.5) 

1 

-0.28 

-0.18 

-0.14 

-0.21 

2 

-0.10 

-0.17 

-0.34 

-0.46 

3 

-0.24 

-0.35 

-0.38 

-0.41 

4 

-0.29 

-0.30 

-0.41 

-0.18 

Overall 

-0.22 

-0.25 

-0.34 

-0.30 

TABLE  2  -  TIMSIM  '93  pertinence-of-engagement  measure  (delay  without  DR  prediction) 


WANTAP 

day 

TWCS 
(-2.5.  0.5) 

FOTC 
(-2.5,  0.5) 

SAG) 

(-2.5,  0.5) 

SAG2 
(-2.5,  0.5) 

1 

-0.33 

-0.34 

-0.24 

-0.30 

2 

-0.27 

-0.24 

-0.42 

-0.48 

3 

-0.27 

-0.36 

-0.43 

-0.48 

4 

-0.39 

-0.43 

-0.48 

-0.22 

Overall 

-0.32 

-0.34 

-0.42 

-0.35 

TABLE  3  -  TIMSIM  '93  pertinence-of-engagement  measure  (delay  with  DR  prediction) 
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Intended-target-opportunity  Measure  -  Summary 


WANTAP 

day 

TWCS 

[-2.5,  0.5] 

FOTC 

[-2.5,  0.5] 

mm 

SAG2 

[-2.5.  0.5] 

1 

0.32 

0.32 

0.34 

0.25 

2 

0.28 

0.17 

0.15 

0.10 

3 

0.10 

-0.10 

-0.05 

•0.12 

4 

0.07 

0.15 

0.19 

0.12 

Overall 

0.17 

0.12 

0.13 

0.08 

TABLE  1  -  TIMSIM  '93  intended-target-opportunity  measure  (no  delay) 


WANTAP 

day 

TWCS 

[-2.5,  0.5] 

FOTC 

[-2.5.  0.5] 

SAG1 

[-2.5,  0.5] 

SAG2 

[-2.5,  0.5] 

1 

-0.29 

-0.22 

-0.18 

-0.24 

2 

-0.11 

-0.17 

-0.34 

-0.46 

3 

-0.25 

-0.37 

-0.40 

-0.43 

4 

•0.30 

-0.32 

-0.41 

-0.19 

Overall 

-0.23 

-0.27 

-0.36 

-0.31 

TABLE  2  -  TIMSIM  '93  intended-target-opportunity  measure  (delay  without  DR  prediction) 


WANTAP 

day 

TWCS 

[-2.5,  0.5] 

FOTC 

[-2.5,  0.5] 

HH 

SAG2 

[-2.5,  0.5] 

1 

-0.35 

-0.36 

-0.29 

-0.32 

2 

•0.28 

-0.24 

-0.43 

-0.48 

3 

-0.29 

-0.38 

-0.45 

-0.49 

4 

-0.40 

-0.44 

-0.49 

-0.23 

Overall 

-0.33 

-0.35 

-0.43 

-0.37 

TABLE  3  -  TIMSIM  ’93  intended-target-opportunity  measure  (delay  with  DR  prediction) 
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Delays  -  Summary 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

46.63 

23.87 

50.15 

43.77 

Friendly 

47.74 

15.57 

25.63 

25.38 

Neutral 

54.75 

19.41 

47.32 

37.66 

Overall 

51.07 

18.79 

37.67 

35.15 

Day  1  -  TIMSIM  '93  mean  differences  between  position  time  and  report  time 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

14.90 

16.09 

24.71 

32.62 

Friendly 

7.94 

15.58 

26.01 

28.81 

Neutral 

15.78 

23.87 

28.71 

44.22 

Overall 

12.95 

19.25 

26.82 

34.54 

Day  2  •  TIMSIM  '93  mean  differences  between  position  time  and  report  time 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

18.66 

21.75 

27.82 

30.07 

Friendly 

8.52 

5.56 

19.35 

16.56 

Neutral 

14.35 

24.90 

34.73 

26.25 

Overall 

13.46 

18.38 

28.94 

25.04 

Day  3  -  TIMSIM  '93  mean  differences  between  position  time  and  report  time 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

14.74 

19.18 

25.58 

23.33 

Friendly 

7.88 

10.85 

16.28 

14.05 

Neutral 

12.09 

28.70 

37.10 

41.37 

Overall 

11.62 

21.58 

28.74 

27.56 

Day  4  •  TIMSIM  '93  mean  differences  between  position  time  and  report  time 
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Positional  Errors  at  Position  Time  (all  cases)  -  Summary 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

2.09 

13.66 

17.87 

109.53 

Friendly 

0.83 

1.11 

1.33 

12.39 

Neutral 

26.41 

12.58 

5.61 

32.76 

Overall 

14.76 

7.49 

5.23 

34.09 

Day  1  •  TIMSIM  '93  mean  differences  between  reported  position  and  GT  position 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

52.76 

110.30 

205.78 

195.51 

Friendly 

0.75 

2.08 

2.27 

2.95 

Neutral 

12.64 

47.91 

56.12 

55.35 

Overall 

15.54 

35.21 

38.76 

34.83 

Day  2  -  TIMSIM  '93  mean  differences  between  reported  position  and  GT  position 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

20.44 

74.09 

70.38 

76.18 

Friendly 

1.06 

2.38 

2.15 

1.98 

Neutral 

14.79 

35.76 

39.09 

144.78 

Overall 

11.73 

31.07 

32.30 

40.29 

Day  3  -  TIMSIM  '93  mean  differences  between  reported  position  and  GT  position 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

9.79 

45.48 

28.08 

60.16 

Friendly 

1.05 

0.95 

2.36 

3.02 

Neutral 

16.91 

18.68 

30.17 

25.46 

Overall 

9.56 

20.12 

21.57 

25.69 

Day  4  -  TIMSIM  '93  mean  differences  between  reported  position  and  GT  position 
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Positional  Errors  at  Position  Time  (outliers  removed)  -  Summary 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

2.09 

1.30 

1.19 

1.03 

Friendly 

0.83 

1.11 

1.33 

1.63 

Neutral 

3.86 

1.40 

0.94 

1.25 

Overall 

2.76 

1.25 

1.20 

1.44 

Day  1  -  TIMSIM  '93  mean  differences  between  reported  position  and  GT  position 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

4.02 

5.41 

4.82 

7.64 

Friendly 

0.75 

1.50 

1.82 

2.48 

Neutral 

0.98 

1.75 

1.19 

1.22 

Overall 

1.36 

1.95 

1.61 

2.31 

Day  2  -  TIMSIM  ‘93  mean  differences  between  reported  position  and  GT  position 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

3.14 

5.47 

5.28 

4.45 

Friendly 

1.06 

1.94 

2.15 

1.98 

Neutral 

1.67 

1.88 

1.74 

0.57 

Overall 

1.82 

2.64 

2.62 

2.95 

Day  3  -  TIMSIM  '93  mean  differences  between  reported  position  and  GT  position 


Allegiance 

TWCS 

FOTC 

SAG1 

SAG2 

Hostile 

2.72 

3.26 

2.90 

3.96 

Friendly 

1.05 

0.95 

2.36 

3.02 

Neutral 

1.49 

2.05 

2.05 

2.76 

Overall 

1.76 

2.00 

2.36 

3.14 

Day  4  -  TIMSIM  '93  mean  differences  between  reported  position  and  GT  position 
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